pub/compliance-scan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feature: IANA update

Browse Source
This commit is contained in:
Heiko
2025-12-19 20:10:39 +01:00
parent f038d6a3fc
commit 753c582010
27 changed files with 1923 additions and 419 deletions
Download Patch File Download Diff File Expand all files Collapse all files

171
docs/detailed-guide.md
View File

@@ -4,14 +4,17 @@ Complete reference for developers and advanced users.
## Core Entry Points
| Component | Path | Purpose |
| --------------- | ------------------------------------ | ------------------------------------- |
| CLI | `src/sslysze_scan/__main__.py` | Command-line interface entry |
| Scanner | `src/sslysze_scan/scanner.py` | SSLyze integration and scan execution |
| Database Writer | `src/sslysze_scan/db/writer.py` | Scan result persistence |
| Reporter | `src/sslysze_scan/reporter/` | Report generation (CSV/MD/reST) |
| Compliance | `src/sslysze_scan/db/compliance.py` | BSI/IANA validation logic |
| Query | `src/sslysze_scan/reporter/query.py` | Database queries using views |
| Component | Path | Purpose |
| --------------- | ------------------------------------------ | ------------------------------------- |
| CLI | `src/sslysze_scan/__main__.py` | Command-line interface entry |
| Scanner | `src/sslysze_scan/scanner.py` | SSLyze integration and scan execution |
| Database Writer | `src/sslysze_scan/db/writer.py` | Scan result persistence |
| Reporter | `src/sslysze_scan/reporter/` | Report generation (CSV/MD/reST) |
| Compliance | `src/sslysze_scan/db/compliance.py` | BSI/IANA validation logic |
| Query | `src/sslysze_scan/reporter/query.py` | Database queries using views |
| IANA Update | `src/sslysze_scan/commands/update_iana.py` | IANA registry updates from web |
| IANA Validator | `src/sslysze_scan/iana_validator.py` | IANA data validation |
| IANA Parser | `src/sslysze_scan/iana_parser.py` | IANA XML parsing utilities |
## Installation
@@ -43,6 +46,8 @@ poetry run compliance-scan report --list
compliance-scan scan <hostname>:<port1>,<port2> [options]
```
Note: SSLyze outputs INFO-level log messages during scanning that cannot be suppressed. These messages are harmless and can be ignored.
| Argument | Required | Description |
| -------------------- | -------- | ---------------------------------------------------------------- |
| `<hostname>:<ports>` | Yes | Target with comma-separated ports. IPv6: `[2001:db8::1]:443,636` |
@@ -78,6 +83,39 @@ compliance-scan report 5 -t csv --output-dir ./reports
compliance-scan report -t rest --output-dir ./docs
```
### Update IANA Command
```
compliance-scan update-iana [-db <path>]
```
| Argument | Required | Description |
| ------------ | -------- | ------------------------------------------------------- |
| `-db <path>` | No | Database file to update (default: compliance_status.db) |
Updates IANA registry data from official sources:
- TLS Parameters: https://www.iana.org/assignments/tls-parameters/tls-parameters.xml
- IKEv2 Parameters: https://www.iana.org/assignments/ikev2-parameters/ikev2-parameters.xml
Default database contains IANA data as of 12/2024.
Examples:
```bash
compliance-scan update-iana
compliance-scan update-iana -db custom.db
```
Update process:
1. Fetches XML from IANA URLs
2. Validates headers against database schema
3. Validates data integrity (value formats, minimum row counts)
4. Calculates diff (added/modified/deleted entries)
5. Updates database in transaction (rollback on error)
6. Logs all changes at INFO level
## Report Formats
### CSV
@@ -230,9 +268,12 @@ src/sslysze_scan/
├── scanner.py # SSLyze integration
├── protocol_loader.py # Port-protocol mapping
├── output.py # Console output
├── iana_parser.py # IANA XML parsing utilities
├── iana_validator.py # IANA data validation
├── commands/
│ ├── scan.py # Scan command handler
│ └── report.py # Report command handler
│ ├── report.py # Report command handler
│ └── update_iana.py # IANA update command handler
├── db/
│ ├── schema.py # Schema version management
│ ├── writer.py # Scan result storage
@@ -241,6 +282,7 @@ src/sslysze_scan/
├── reporter/
│ ├── query.py # Database queries (uses views)
│ ├── csv_export.py # CSV generation
│ ├── csv_utils.py # CSV utilities (exporter class)
│ ├── markdown_export.py # Markdown generation
│ ├── rst_export.py # reST generation
│ └── template_utils.py # Shared utilities
@@ -249,7 +291,16 @@ src/sslysze_scan/
│ └── report.reST.j2 # reST template
└── data/
├── crypto_standards.db # Template DB (IANA/BSI + schema)
├── iana_parse.json # IANA XML source URLs and registry config
└── protocols.csv # Port-protocol mapping
tests/
├── fixtures/
│ ├── iana_xml/ # Minimal XML test fixtures
│ └── test_scan.db # Test database
├── test_iana_validator.py # IANA validation tests (25 tests)
├── test_iana_parse.py # IANA XML parsing tests (20 tests)
└── test_iana_update.py # IANA update logic tests (13 tests)
```
## Key Functions
@@ -284,6 +335,7 @@ src/sslysze_scan/
| `get_scan_data(db_path, scan_id)` | `reporter/query.py` | Get complete scan data using views |
| `get_scan_metadata(db_path, scan_id)` | `reporter/query.py` | Get scan metadata only |
| `list_scans(db_path)` | `reporter/query.py` | List all scans in database |
| `has_tls_support(port_data)` | `reporter/query.py` | Check if port has TLS support |
### Report Generation
@@ -292,10 +344,101 @@ src/sslysze_scan/
| `generate_csv_reports(db_path, scan_id, output_dir)` | `reporter/csv_export.py` | Generate all CSV files |
| `generate_markdown_report(db_path, scan_id, output)` | `reporter/markdown_export.py` | Generate Markdown report |
| `generate_rest_report(db_path, scan_id, output, output_dir)` | `reporter/rst_export.py` | Generate reStructuredText report |
| `_get_headers(db_path, export_type)` | `reporter/csv_export.py` | Load CSV headers from database |
| `build_template_context(data)` | `reporter/template_utils.py` | Prepare Jinja2 template context |
| `generate_report_id(metadata)` | `reporter/template_utils.py` | Generate report ID (YYYYMMDD_scanid) |
### IANA Update and Validation
| Function | Module | Purpose |
| ------------------------------------------------ | ------------------------- | ---------------------------------------- |
| `handle_update_iana_command(args)` | `commands/update_iana.py` | Main update command handler |
| `fetch_xml_from_url(url)` | `commands/update_iana.py` | Fetch XML from IANA URL |
| `calculate_diff(old_rows, new_rows)` | `commands/update_iana.py` | Calculate added/modified/deleted entries |
| `process_registry_with_validation(...)` | `commands/update_iana.py` | Process and validate single registry |
| `validate_headers(table_name, headers, db_conn)` | `iana_validator.py` | Validate headers match database schema |
| `validate_registry_data(table_name, rows)` | `iana_validator.py` | Validate complete registry data |
| `validate_cipher_suite_row(row)` | `iana_validator.py` | Validate single cipher suite record |
| `validate_supported_groups_row(row)` | `iana_validator.py` | Validate single supported group record |
| `normalize_header(header)` | `iana_validator.py` | Normalize header to DB column format |
| `get_min_rows(table_name)` | `iana_validator.py` | Get minimum expected rows for table |
| `extract_updated_date(xml_content)` | `iana_parser.py` | Extract date from XML `<updated>` tag |
| `parse_xml_with_namespace_support(xml_path)` | `iana_parser.py` | Parse XML with IANA namespace detection |
| `find_registry(root, registry_id, ns)` | `iana_parser.py` | Find registry element by ID |
| `extract_field_value(record, header, ns)` | `iana_parser.py` | Extract field value from XML record |
## IANA Data Update Process
Configuration file: `src/sslysze_scan/data/iana_parse.json`
Structure:
```json
{
"https://www.iana.org/assignments/tls-parameters/tls-parameters.xml": [
["registry_id", "output_filename.csv", ["Header1", "Header2", "..."]]
]
}
```
Validation rules:
1. Headers must match database schema (case-insensitive, `/` → `_`)
2. Minimum row counts per table (50 for cipher suites, 10 for groups, 5 for small tables)
3. Value format validation (0x prefix for hex values, numeric for groups)
4. Recommended field must be Y, N, or D
Error handling:
- Validation failure: Rollback transaction, display error with hint to open issue
- Network error: Abort with error message
- XML structure change: Validation catches and aborts
Logging output:
```
INFO: Fetching https://www.iana.org/assignments/tls-parameters/tls-parameters.xml
INFO: XML data date: 2025-12-03
INFO: iana_tls_cipher_suites: 448 rows (2 added, 1 modified, 0 deleted)
INFO: Successfully updated 11 registries (1310 total rows)
```
## Version Management
Version is maintained in `pyproject.toml` only:
```toml
[project]
version = "0.1.0"
```
Runtime access via `importlib.metadata`:
```python
from sslysze_scan import __version__
print(__version__) # "0.1.0"
```
## Development
Run tests:
```bash
poetry run pytest
poetry run pytest tests/test_iana_validator.py -v
```
Update IANA template database:
```bash
python3 -m sslysze_scan.iana_parser
```
Code style:
- PEP 8 compliant
- Max line length: 90 characters
- Ruff for linting and formatting
## SQL Query Examples
All queries use optimized views for performance.
@@ -382,11 +525,13 @@ poetry run pytest tests/ -v
- `tests/conftest.py`: Fixtures with test_db, test_db_path
- `tests/fixtures/test_scan.db`: Real scan data (Scan 1: dc.validation.lan:443,636)
- `tests/test_csv_export.py`: 11 CSV export tests
- `tests/test_template_utils.py`: 3 template utility tests
- `tests/test_compliance.py`: 2 compliance tests
- `tests/test_template_utils.py`: 2 template utility tests
- `tests/test_cli.py`: 3 CLI parsing tests
- `tests/test_iana_validator.py`: 20 IANA validation tests
- `tests/test_iana_parse.py`: 14 IANA parsing tests
- `tests/test_iana_update.py`: 13 IANA update tests
**Total:** 19 tests
**Total:** 63 tests
**Test database setup:**