feat: implement contrast improvements, pre/post dedup MSDP chart, Azure/OCI PDF label, and live troubleshooting crawler
This commit is contained in:
Silas Brito
@@ -1,103 +1,124 @@
|
||||
Markdown
|
||||
# Software Requirements Specification (SRS) - v2.0
|
||||
## Project Name: NetBackup Log Insights & Action Tracker (Docker & Gitea Edition)
|
||||
## Target Environment: Docker Container for VPS Deployment
|
||||
## IDE Target: Antigravity IDE
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary & New Objectives
|
||||
The objective is to build a modern, containerized backup management dashboard using Python, Streamlit, and SQLite. The system processes NetBackup "Job Summary" CSV exports, segregates infrastructure cloud zones, cross-references transient errors with automatic successes, tracks manual operational actions, and exports standardized PDF mitigation reports.
|
||||
|
||||
---
|
||||
|
||||
## 2. Infrastructure & Portability Guardrails (Docker Shift)
|
||||
|
||||
### 2.1 Containerization Deployment
|
||||
* **Base Image:** `python:3.11-slim` (to keep the cloud footprint minimal).
|
||||
* **Port Configuration:** Expose port `8501` internally, mapped via Docker Compose.
|
||||
* **Volume Mapping:** Data persistence must be maintained by mapping a host directory to the container’s internal storage path (`/app/data`).
|
||||
|
||||
### 2.2 Advanced Database Architecture & Persistence
|
||||
* **Database File:** `/app/data/nbu_insights.db` (Mapped volume).
|
||||
* **Historical Traceability:** Data uploaded via CSV must be completely persisted. If a new CSV is processed, the system must perform an `UPSERT` operation based on the unique key `(Job Id, Client, Policy)` to preserve historical runs and ensure that previously typed mitigation actions are never overwritten.
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS processed_files (
|
||||
file_hash TEXT PRIMARY KEY,
|
||||
file_name TEXT,
|
||||
upload_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS backup_jobs (
|
||||
job_id INTEGER PRIMARY KEY,
|
||||
client TEXT,
|
||||
policy TEXT,
|
||||
type TEXT,
|
||||
exit_code INTEGER,
|
||||
start_time TIMESTAMP,
|
||||
finish_time TIMESTAMP,
|
||||
duration_secs INTEGER,
|
||||
mbytes REAL,
|
||||
files_count INTEGER,
|
||||
primary_server TEXT,
|
||||
media_server TEXT,
|
||||
is_rerun_success INTEGER DEFAULT 0
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS job_actions (
|
||||
job_id INTEGER PRIMARY KEY,
|
||||
action_taken TEXT,
|
||||
status TEXT DEFAULT 'Pendente', -- Enum: 'Pendente', 'Em Progresso', 'Resolvido'
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
FOREIGN KEY(job_id) REFERENCES backup_jobs(job_id)
|
||||
);
|
||||
3. UI/UX Design & Branding Strategy (CXP-Inspired Dark Tech)
|
||||
Primary Background: #0B0F19 (Deep Midnight Blue)
|
||||
|
||||
Card / Container Background: #1E2640 (Slate Corporate Blue)
|
||||
|
||||
Primary Accent: #00D2FF (Vibrant Cyan)
|
||||
|
||||
Typography Hierarchy: Clean, highly visible tech elements with colored status badges (#00C853 for active success, #FF4B4B for unmitigated failure).
|
||||
|
||||
4. Full Functional Requirements & Data Insights Engine
|
||||
4.1 Ingestion & Cleansing Pipeline
|
||||
CSV Offset: Dynamically strip metadata row 0 (###################### TABLE...).
|
||||
|
||||
Metric Extraction: Auto-calculate MSDP Deduplication Ratio and Savings. Show globally and separate per Primary Server.
|
||||
|
||||
4.2 Multi-Cloud Tenant Views (Sidebar Selector)
|
||||
Consolidated View: Combined global metrics.
|
||||
|
||||
Azure Infrastructure Zone: Filters data where Primary Server == 'srvpalcvnbu01.elo.corp'.
|
||||
|
||||
OCI Infrastructure Zone: Filters data where Primary Server == 'srvpalcocinbupri01.elo.corp'.
|
||||
|
||||
4.3 Advanced Rerun & Target Resolution Analytics
|
||||
Automated Clearing: If a job fails (Exit Code > 1) but the same Client and Policy have a subsequent successful job (Exit Code 0 or 1) inside the logs, flag it automatically as [✓ Reexecutado com Sucesso].
|
||||
|
||||
State Filtering: Create a dedicated filter to view:
|
||||
|
||||
Todos os Erros
|
||||
|
||||
Erros Sem Tratativa / Pendentes
|
||||
|
||||
Erros Corrigidos Automatizados (Reexecutados)
|
||||
|
||||
4.4 Action Register & PDF Mitigation Report Engine
|
||||
Form Action: Input text fields for engineers to log infrastructure actions (e.g., "Adjusted firewall rules on port 1556", "Expanded snapshot window").
|
||||
|
||||
PDF Generation Engine: Integrating the fpdf2 or reportlab library. Clicking "Exportar Plano de Mitigação PDF" triggers a compiled file containing a professional summary header, performance graphs metrics, and a clean table of all logged actions and resolutions.
|
||||
|
||||
5. Repository File Layout
|
||||
Plaintext
|
||||
netbackup-insights/
|
||||
│
|
||||
├── app.py # Main UI Orchestrator
|
||||
├── database.py # SQLite Connection & Upsert Engine
|
||||
├── parser.py # Pandas Data Cleansing and Ratio Math
|
||||
├── report_gen.py # PDF Layout Compilation Engine
|
||||
├── requirements.txt # Python Dependencies
|
||||
├── Dockerfile # App Container Specification
|
||||
└── docker-compose.yml # Multi-environment VPS deployment spec
|
||||
Markdown
|
||||
# Software Requirements Specification (SRS) - v2.0
|
||||
## Project Name: NetBackup Log Insights & Action Tracker (Docker & Gitea Edition)
|
||||
## Target Environment: Docker Container for VPS Deployment
|
||||
## IDE Target: Antigravity IDE
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary & New Objectives
|
||||
The objective is to build a modern, containerized backup management dashboard using Python, Streamlit, and SQLite. The system processes NetBackup "Job Summary" CSV exports, segregates infrastructure cloud zones, cross-references transient errors with automatic successes, tracks manual operational actions, and exports standardized PDF mitigation reports.
|
||||
|
||||
---
|
||||
|
||||
## 2. Infrastructure & Portability Guardrails (Docker Shift)
|
||||
|
||||
### 2.1 Containerization Deployment
|
||||
* **Base Image:** `python:3.11-slim` (to keep the cloud footprint minimal).
|
||||
* **Port Configuration:** Expose port `8501` internally, mapped via Docker Compose.
|
||||
* **Volume Mapping:** Data persistence must be maintained by mapping a host directory to the container’s internal storage path (`/app/data`).
|
||||
|
||||
### 2.2 Advanced Database Architecture & Persistence
|
||||
* **Database File:** `/app/data/nbu_insights.db` (Mapped volume).
|
||||
* **Historical Traceability:** Data uploaded via CSV must be completely persisted. If a new CSV is processed, the system must perform an `UPSERT` operation based on the unique key `(Job Id, Client, Policy)` to preserve historical runs and ensure that previously typed mitigation actions are never overwritten.
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS processed_files (
|
||||
file_hash TEXT PRIMARY KEY,
|
||||
file_name TEXT,
|
||||
upload_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS backup_jobs (
|
||||
job_id INTEGER PRIMARY KEY,
|
||||
client TEXT,
|
||||
policy TEXT,
|
||||
type TEXT,
|
||||
exit_code INTEGER,
|
||||
start_time TIMESTAMP,
|
||||
finish_time TIMESTAMP,
|
||||
duration_secs INTEGER,
|
||||
mbytes REAL,
|
||||
files_count INTEGER,
|
||||
primary_server TEXT,
|
||||
media_server TEXT,
|
||||
is_rerun_success INTEGER DEFAULT 0
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS job_actions (
|
||||
job_id INTEGER PRIMARY KEY,
|
||||
action_taken TEXT,
|
||||
status TEXT DEFAULT 'Pendente', -- Enum: 'Pendente', 'Em Progresso', 'Resolvido'
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
FOREIGN KEY(job_id) REFERENCES backup_jobs(job_id)
|
||||
);
|
||||
3. UI/UX Design & Branding Strategy (CXP-Inspired Dark Tech)
|
||||
Primary Background: #0B0F19 (Deep Midnight Blue)
|
||||
|
||||
Card / Container Background: #1E2640 (Slate Corporate Blue)
|
||||
|
||||
Primary Accent: #00D2FF (Vibrant Cyan)
|
||||
|
||||
Typography Hierarchy: Clean, highly visible tech elements with colored status badges (#00C853 for active success, #FF4B4B for unmitigated failure).
|
||||
|
||||
4. Full Functional Requirements & Data Insights Engine
|
||||
4.1 Ingestion & Cleansing Pipeline
|
||||
CSV Offset: Dynamically strip metadata row 0 (###################### TABLE...).
|
||||
|
||||
Metric Extraction: Auto-calculate MSDP Deduplication Ratio and Savings. Show globally and separate per Primary Server.
|
||||
|
||||
4.2 Multi-Cloud Tenant Views (Sidebar Selector)
|
||||
Consolidated View: Combined global metrics.
|
||||
|
||||
Azure Infrastructure Zone: Filters data where Primary Server == 'srvpalcvnbu01.elo.corp'.
|
||||
|
||||
OCI Infrastructure Zone: Filters data where Primary Server == 'srvpalcocinbupri01.elo.corp'.
|
||||
|
||||
4.3 Advanced Rerun & Target Resolution Analytics
|
||||
Automated Clearing: If a job fails (Exit Code > 1) but the same Client and Policy have a subsequent successful job (Exit Code 0 or 1) inside the logs, flag it automatically as [✓ Reexecutado com Sucesso].
|
||||
|
||||
State Filtering: Create a dedicated filter to view:
|
||||
|
||||
Todos os Erros
|
||||
|
||||
Erros Sem Tratativa / Pendentes
|
||||
|
||||
Erros Corrigidos Automatizados (Reexecutados)
|
||||
|
||||
4.4 Action Register & PDF Mitigation Report Engine
|
||||
Form Action: Input text fields for engineers to log infrastructure actions (e.g., "Adjusted firewall rules on port 1556", "Expanded snapshot window").
|
||||
|
||||
PDF Generation Engine: Integrating the fpdf2 or reportlab library. Clicking "Exportar Plano de Mitigação PDF" triggers a compiled file containing a professional summary header, performance graphs metrics, and a clean table of all logged actions and resolutions.
|
||||
|
||||
5. Repository File Layout
|
||||
Plaintext
|
||||
netbackup-insights/
|
||||
│
|
||||
├── app.py # Main UI Orchestrator
|
||||
├── database.py # SQLite Connection & Upsert Engine
|
||||
├── parser.py # Pandas Data Cleansing and Ratio Math
|
||||
├── report_gen.py # PDF Layout Compilation Engine
|
||||
├── requirements.txt # Python Dependencies
|
||||
├── Dockerfile # App Container Specification
|
||||
└── docker-compose.yml # Multi-environment VPS deployment spec
|
||||
|
||||
---
|
||||
|
||||
## 6. Enhanced Web Features (v2.1)
|
||||
|
||||
### 6.1 Contrast & Accessibility Standardization
|
||||
* **Backgrounds:** Maintain `#0B0F19` deep blue background for pages. Use a distinct slate-blue `#161E35` for metric cards and inputs to separate sections visually.
|
||||
* **Text Contrast:** Ensure all metric values use `#FFFFFF`. Use `#F1F5F9` for secondary headers/body text and `#CBD5E1` for metric labels.
|
||||
* **Badges:** Success badge uses brighter `#00E676` green, warning badge uses `#FFD600` yellow.
|
||||
|
||||
### 6.2 MSDP Volumetry Grouped Chart
|
||||
* Show a double bar chart for client data consumption comparison.
|
||||
* **Metrics:** Side-by-side comparison of **Pré-Dedup (Volume Bruto)** in `#3B82F6` blue and **Pós-Dedup (Volume Gravado)** in `#10B981` green for the top 7 clients.
|
||||
|
||||
### 6.3 PDF Cloud Infrastructure Flag
|
||||
* The PDF report table must contain a dedicated **Infra/Cloud** column.
|
||||
* **Rules:** If `Primary Server` contains `srvpalcvnbu01`, classify as `Azure`. If it contains `srvpalcocinbupri01`, classify as `OCI`. Otherwise, label as `Outro`.
|
||||
|
||||
### 6.4 Live Internet Troubleshooting & Diagnostic Engine
|
||||
* For failed jobs that are not cleared by re-runs (`is_rerun_success == 0`), the UI must resolve troubleshooting steps dynamically.
|
||||
* **Mechanism:** Fall back to a local database dictionary (for status codes 2, 25, 26, 57, 58, 96, 156), and perform a real-time HTTP search query against `html.duckduckgo.com` to fetch supplemental resolution notes online.
|
||||
Reference in New Issue
Block a user