Gateway: Add SOVD bulk-data endpoints and fault detail response#165
Merged
Gateway: Add SOVD bulk-data endpoints and fault detail response#165
Conversation
…rEntity definitions Add new ROS2 message definitions for SOVD-compliant fault environment data: - ExtendedDataRecords.msg: first/last occurrence timestamps - Snapshot.msg: unified freeze frame and rosbag snapshot type - EnvironmentData.msg: container for snapshots in fault response - GetFault.srv: NEW service for single fault lookup with environment_data - ListFaultsForEntity.srv: service to query faults by entity
Add UUID generation for rosbag files and index for UUID-based lookup: - FaultStorage::generate_uuid() for UUID v4 generation - RosbagFileInfo::bulk_data_id field for globally unique identifier - FaultStorage::get_rosbag_by_id() for lookup by UUID - FaultStorage::get_rosbag_path() for file path retrieval - FaultStorage::get_rosbags_for_entity() for entity-filtered queries - rosbag_by_id_ index for O(1) UUID lookups in InMemoryFaultStorage - SQLite schema migration for existing databases Required for SOVD bulk-data download pattern. @verifies REQ_INTEROP_073
Implement GetFault service handler that returns: - Fault data with basic info and status - ExtendedDataRecords with first/last occurrence timestamps - Snapshots including freeze frames and rosbag metadata Add unit tests for GetFault service responses.
Implement ListFaultsForEntity service for querying faults by entity: - Add get_all_faults() method to FaultStorage interface and implementations - Add matches_entity() helper for FQN suffix matching - Add list_faults_for_entity service to FaultManagerNode Add unit tests for new functionality including: - GetAllFaults storage tests - ListFaultsForEntity service tests - matches_entity helper tests
Add utility functions for parsing entity paths from HTTP request URLs:
- EntityPathInfo struct with type, entity_id, resource_path, parent_id
- parse_entity_path() extracts entity info from /api/v1/{type}/{id}/... URLs
- extract_bulk_data_category() gets bulk-data category from path
- extract_bulk_data_id() gets bulk-data item ID from path
Supports nested entities (subareas, subcomponents) with parent tracking.
Includes 20 unit tests covering all parsing scenarios.
Add build_sovd_fault_response() for SOVD-compliant responses: - 'item' wrapper with fault details and status object - 'environment_data' with extended_data_records and snapshots - 'x-medkit' extensions with occurrence_count, severity_label - Rosbag snapshots include bulk_data_uri (absolute path) - Primary value extraction for common ROS message types Add FaultManager::get_fault_with_env() for retrieving fault with environment data via GetFault service. Update handle_get_fault to use entity path from URL for bulk_data_uri generation. Includes 12 unit tests covering SOVD response building. @verifies REQ_INTEROP_013
Add BulkDataHandlers class with:
- handle_list_categories: GET {entity}/bulk-data
- handle_list_descriptors: GET {entity}/bulk-data/{category}
- handle_download: GET {entity}/bulk-data/{category}/{id}
Supports all entity path patterns:
- /apps, /components, /areas, /functions
- /areas/{id}/subareas, /components/{id}/subcomponents
Security: Validates rosbag belongs to requested entity.
CORS: Exposes Content-Disposition header for frontend.
@verifies REQ_INTEROP_071
@verifies REQ_INTEROP_072
@verifies REQ_INTEROP_073
REQ_INTEROP_072, REQ_INTEROP_073
- Remove /faults/{code}/snapshots (system-wide)
- Remove /faults/{code}/snapshots/bag (rosbag download)
- Remove entity-scoped snapshot endpoints for components, apps, areas, functions
- Remove handler declarations from fault_handlers.hpp
- Remove handler implementations from fault_handlers.cpp
- Remove unused helper functions (sanitize_filename, generate_timestamp, validate_fault_code)
- Remove unused includes (filesystem, fstream, iomanip)
Snapshots are now inline in fault responses (environment_data).
Rosbag downloads use bulk-data endpoint pattern (GET /{entity}/bulk-data/rosbags/{fault_code}).
Add comprehensive integration tests: - Bulk-data category listing for all entity types - BulkDataDescriptor structure validation - Rosbag download with headers verification - Security check for cross-entity access - Nested entity path support - SOVD-compliant fault response structure - Legacy endpoints removal verification @verifies REQ_INTEROP_071 @verifies REQ_INTEROP_072 @verifies REQ_INTEROP_073 @verifies REQ_INTEROP_013
- Add resolve_rosbag_file_path() to handle rosbag2 directory structure - Fix MIME type from application/gzip to application/x-sqlite3 for db3 files - Update unit tests for correct MIME type expectations - Update integration tests for SQLite format validation
Update documentation for SOVD bulk-data feature: - Add Bulk Data Endpoints section to REST API reference - Update fault response with environment_data and SOVD status - Rewrite snapshots tutorial for new API pattern - Update changelog with breaking changes - Mark REQ_INTEROP_071-073 as verified - Update REQ_INTEROP_013 for fault environment data - Remove REQ_INTEROP_088 (legacy snapshots endpoint) Documents breaking changes and migration path.
Fix spelling typo propagated through the entire stack: - ExtendedDataRecords.msg: first_occurence_ns → first_occurrence_ns - fault_manager_node.cpp: field access updated - fault_handlers.cpp: JSON keys updated - All test files updated to match This is a breaking change in the ROS message API, fixed now before the bulk-data feature is merged to main.
P0: Replace memory-loading file transfer with chunked content provider using httplib::set_content_provider() with 64KB chunks. Rosbag files can be hundreds of MB to multiple GB. P0: Add GetRosbags batch service to eliminate N+1 service calls in handle_list_descriptors(). Uses existing get_rosbags_for_entity() storage method via a new ROS 2 service. P1: Add Content-Disposition and Content-Length to global CORS Access-Control-Expose-Headers in both RESTServer and HandlerContext. P0: Document mutex locking contract on FaultManager::service_mutex_ to prevent future deadlock risks in get_fault() delegation.
Consolidate duplicate timestamp formatting (to_iso8601_ns in fault_handlers.cpp and format_timestamp_ns in bulkdata_handlers.cpp) into a shared inline function in http_utils.hpp. Add null check for gmtime_r return value — previously undefined behavior on negative or invalid timestamps. Add unit tests for the shared utility covering valid timestamps, epoch, milliseconds, and negative timestamp fallback.
Apply consistent naming convention for ROS 2 service definitions: - Get* = retrieve single item by key (GetFault, GetRosbag) - List* = query collection with filters (ListFaults, ListRosbags, ListFaultsForEntity) Renames .srv files, all C++ types/includes/variables/handlers, Python imports, service endpoint names (~/list_faults, ~/list_rosbags), storage method names, and documentation references. BREAKING: Service endpoint names changed: ~/get_faults → ~/list_faults ~/get_rosbags → ~/list_rosbags
… routing) - Make snapshot/rosbag capture async in handle_report_fault to avoid blocking the single-threaded executor for seconds during on-demand capture. This prevents gateway lag and connection timeouts. - Split single service_mutex_ into per-client mutexes so read operations (list_faults, get_fault) are not blocked during slow write operations (report_fault with snapshot capture). - Fix bulk-data download routing: build_sovd_fault_response was using rosbag UUID (s.bulk_data_id) in bulk_data_uri, but the download handler expects fault_code. Changed to use fault.fault_code, consistent with handle_list_descriptors.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR implements SOVD-compliant bulk-data endpoints for downloading rosbag recordings and restructures fault responses to include inline environment data with snapshots. The changes replace legacy snapshot endpoints with SOVD patterns using UUID-based bulk-data identifiers.
Changes:
- Implemented SOVD bulk-data endpoints for rosbag downloads across all entity types
- Restructured fault responses to include inline environment_data with freeze_frame and rosbag snapshots
- Added UUID identifiers for rosbag files to enable bulk-data download pattern
Reviewed changes
Copilot reviewed 54 out of 55 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/ros2_medkit_msgs/srv/ListRosbags.srv | Added batch service for retrieving rosbag metadata by entity |
| src/ros2_medkit_msgs/srv/ListFaultsForEntity.srv | Added service for filtering faults by entity |
| src/ros2_medkit_msgs/srv/GetFault.srv | New service to retrieve single fault with environment data |
| src/ros2_medkit_msgs/msg/Snapshot.msg | Unified snapshot type supporting freeze_frame and rosbag variants |
| src/ros2_medkit_msgs/msg/EnvironmentData.msg | SOVD environment data container with snapshots |
| src/ros2_medkit_gateway/src/http/handlers/bulkdata_handlers.cpp | Bulk-data handler implementation for listing and downloading rosbags |
| src/ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp | Updated to build SOVD-compliant fault responses with environment data |
| src/ros2_medkit_gateway/src/http/entity_path_utils.cpp | Entity path parsing utilities for bulk-data endpoints |
| src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp | Added UUID support and entity-filtered rosbag queries |
| src/ros2_medkit_fault_manager/src/fault_manager_node.cpp | Implemented GetFault and ListRosbags services with async snapshot capture |
bburda
changed the title
…x typos - Update 3 failing tests to expect fault_code instead of UUID in bulk_data_uri - Rename fc/sc/rc to fault_code/snapshot_capture_ptr/rosbag_capture_ptr - Fix 'occurence' -> 'occurrence' typos in docs
- Fix RST title underlines in api/messages.rst (ListFaults.srv) and design/ros2_medkit_fault_manager/index.rst (~/list_faults) that were shorter than the title text - Add missing REQ_INTEROP_088 requirement definition in specs/faults.rst for snapshot and rosbag capture on fault confirmation, referenced by multiple verification tests
Pass automatically_add_to_executor_with_node=false when creating the local callback group in capture_topic_on_demand(). Without this, the callback group is automatically added to the node's main executor (via rclcpp::spin), and then add_callback_group() tries to add it to the local SingleThreadedExecutor, throwing 'Callback group has already been added to an executor'.
mfaferek93
reviewed
Feb 6, 2026
src/ros2_medkit_gateway/src/http/handlers/bulkdata_handlers.cpp
Outdated
Show resolved
Hide resolved
src/ros2_medkit_gateway/src/http/handlers/bulkdata_handlers.cpp
Outdated
Show resolved
Hide resolved
- Make generate_uuid() thread-safe with thread_local instead of static - Convert snapshot/rosbag capture to shared_ptr for safe detached thread capture - Replace LIKE with json_each() for proper JSON array querying in SQLite - Use validate_entity_for_route() in bulkdata handlers for route-type consistency - Replace list_faults ownership check with targeted get_fault lookup
- item.code not item.faultCode - extended_data_records is an object (not list) with first/last_occurrence - snapshots use type/name fields, not category/id/timestamp
The HTTP layer uses fault_code as the bulk-data identifier (1:1 mapping), making the UUID generation and UUID-based lookup completely unused: - Remove FaultStorage::generate_uuid() and related includes - Remove get_rosbag_by_id() / get_rosbag_path() from base class and impls - Remove bulk_data_id field from RosbagFileInfo struct - Remove bulk_data_id column from SQLite schema and migration code - Remove rosbag_by_id_ index map from InMemoryFaultStorage - Remove UUID-related tests (7 tests) - Set Snapshot.msg bulk_data_id to fault_code instead of UUID - Update gateway tests to use fault_code for bulk_data_id field -279 lines removed (YAGNI)
test_104_set_configuration was picking the first config parameter which could be a read-only qos_override. Now iterates through params, checks the read_only descriptor flag, and also handles unexpected 403 responses by trying the next parameter.
mfaferek93
approved these changes
Feb 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Summary
Implements SOVD-compliant bulk-data endpoints for downloading rosbag recordings and restructures fault responses to include inline environment data with snapshots. The changes replace legacy snapshot endpoints with SOVD patterns using UUID-based bulk-data identifiers.
Changes:
Issue
Link the related issue (required):
Type
Testing
How was this tested / how should reviewers verify it?
Demos + colcon tests
Checklist