Skip to content

Fix: Scan FSH files and additional directories for Logical Models and ValueSets #183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Copilot wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Fix: Scan FSH files and additional directories for Logical Models and ValueSets #183

Copilot wants to merge 3 commits into from

Conversation

Copy link
Contributor

Copilot AI commented Oct 9, 2025 •
edited
Loading

✅ Issue Resolved: LM and valusets not found

Summary

Successfully fixed the update_sushi_config.py script to scan FSH files and additional directories for Logical Models and ValueSets.

Recent Changes

  • Extracted FSH parsing regex patterns into module-level constants for improved maintainability
    • LOGICAL_PATTERN - Pattern for matching Logical Model declarations
    • VALUESET_PATTERN - Pattern for matching ValueSet declarations
    • TITLE_PATTERN - Pattern for extracting Title metadata
    • DESCRIPTION_PATTERN - Pattern for extracting Description metadata
  • Updated both parsing functions to use shared constants, eliminating duplication

Original Implementation

  • Added parse_fsh_file_for_logical_model() function to parse FSH Logical Model definitions
  • Added parse_fsh_file_for_valueset() function to parse FSH ValueSet definitions
  • Extended scan_for_valuesets_and_create_placeholders() to scan 4 additional directories:
    • input/fsh/models (FSH Logical Models)
    • input/fsh/valuesets (FSH ValueSets)
    • input/models (JSON StructureDefinitions)
    • input/vocabulary (JSON ValueSets)
  • Maintained backward compatibility with existing scanning of:
    • fsh-generated/resources (SUSHI output)
    • input/resources (static JSON)

Results

Before (Issue)

Found 0 ValueSets
Found 0 Logical Models
Created 0 placeholder files

After (Fixed)

Found 6 ValueSets
Found 16 Logical Models
Created 21 placeholder files

Resources Found

  • 16 Logical Models: DAK, GenericPersona, HealthInterventions, TestScenario, NonFunctionalRequirement, BusinessProcessWorkflow, ProgramIndicator, HealthInterventionsSource, DecisionSupportLogic, Persona, DublinCore, SushiConfigLogicalModel, FunctionalRequirement, CoreDataElement, Requirements, UserScenario
  • 6 ValueSets: CDSCv1, SGPersonaTypesVS, ISCO08ValueSet, DecisionTableActions, DocumentationSection, CDHIv1

Testing

  • ✅ All 16 Logical Models found from FSH files
  • ✅ All 6 ValueSets found from FSH files
  • ✅ Placeholder markdown files created correctly
  • ✅ Duplicate detection works across all sources
  • ✅ Existing custom documentation preserved
  • ✅ Edge cases handled (missing directories, etc.)
  • ✅ QA report shows 0 errors
  • ✅ Regex pattern refactoring verified with unit tests
Original prompt

This section details on the original issue you should resolve

<issue_title>LM and valusets not found</issue_title>
<issue_description>```
Run echo "Preparing DAK environment with configuration scripts..."
Preparing DAK environment with configuration scripts...
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (6.0.1)
Running generate_dak_from_sushi.py...
DAK configuration already exists at dak.json, skipping generation
✅ DAK configuration processing completed
Running update_sushi_config.py...
This is the smart-base repository (id: smart.who.int.base)
dak-api.md already exists with proper content
Scanning input/resources for FHIR resources...
Placeholder generation completed:
Found 0 ValueSets
Found 0 Logical Models
Created 0 placeholder files

I would have expected to find the LMs and valussets in smart-base under input/fsh/models and input/fsh/valusets.  please ensure both the input/fsh is scanned for fsh files and input/models and input/valusets is scanned for fhir json </issue_description>

## Comments on the Issue (you are @copilot in this section)

<comments>
</comments>

Fixes #179

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@litlfred
Add FSH and additional directory scanning to update_sushi_config.py
ebe439e 
Co-authored-by: litlfred <662242+litlfred@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix missing LMs and valueSets in smart-base Fix: Scan FSH files and additional directories for Logical Models and ValueSets Oct 9, 2025
Copilot AI requested a review from litlfred October 9, 2025 18:02
@litlfred litlfred marked this pull request as ready for review October 9, 2025 20:41
Copilot AI review requested due to automatic review settings October 9, 2025 20:41
@github-actions
Copy link

github-actions bot commented Oct 9, 2025 •
edited
Loading

❌ Deployment Failed

Branch: copilot/fix-lm-and-valuest-issue
Commit: 66a51aa

🔍 View Build Log

Deployment failed. Please check the build log for details.

@github-actions
Copy link

github-actions bot commented Oct 9, 2025 •
edited
Loading

✅ Deployment Successful

Branch: copilot/fix-lm-and-valuest-issue
Commit: ebe439e
Deployed to: https://worldhealthorganization.github.io/smart-base/branches/fix-lm-and-valuest-issue/

🌐 Open Preview
🔍 View Build Log

Deployment completed successfully! The site is now live at the URL above.

Copilot AI reviewed Oct 9, 2025
View reviewed changes
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes the update_sushi_config.py script to properly discover Logical Models and ValueSets from FSH source files and additional JSON directories. The script was previously only scanning compiled JSON resources, missing the 16 Logical Models and 6 ValueSets defined in FSH files throughout the repository.

  • Enhanced the script to parse FSH files directly using regex patterns to extract resource metadata
  • Extended directory scanning from 2 to 6 locations including FSH source directories and alternative JSON paths
  • Added comprehensive duplicate detection and QA reporting for all discovered resources

Reviewed Changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 6 comments.

File Description
input/scripts/update_sushi_config.py Core enhancement adding FSH parsing functions and extending directory scanning to discover all Logical Models and ValueSets
input/pagecontent/ValueSet-*.md Generated placeholder markdown files for discovered ValueSets with proper DAK API integration markers
input/pagecontent/StructureDefinition-*.md Generated placeholder markdown files for discovered Logical Models with proper DAK API integration markers

input/scripts/update_sushi_config.py Outdated
content = f.read()

# Look for Logical: declaration
logical_match = re.search(r'^Logical:\s+(\S+)', content, re.MULTILINE)
Copy link

Copilot AI Oct 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider extracting the regex patterns into module-level constants to improve maintainability and reduce duplication. For example: LOGICAL_PATTERN = r'^Logical:\s+(\S+)' and VALUESET_PATTERN = r'^ValueSet:\s+(\S+)'

Copilot uses AI. Check for mistakes.
Copy link
Collaborator

@litlfred litlfred Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot apply changes based on this feedback

Copy link
Contributor Author

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extracted all FSH parsing regex patterns into module-level constants as suggested. The patterns are now defined once at the top of the module and reused in both parsing functions. Changes verified with tests. (80d0873)

input/scripts/update_sushi_config.py Outdated
content = f.read()

# Look for ValueSet: declaration
valueset_match = re.search(r'^ValueSet:\s+(\S+)', content, re.MULTILINE)
Copy link

Copilot AI Oct 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider extracting the regex patterns into module-level constants to improve maintainability and reduce duplication. For example: LOGICAL_PATTERN = r'^Logical:\s+(\S+)' and VALUESET_PATTERN = r'^ValueSet:\s+(\S+)'

Copilot uses AI. Check for mistakes.
Copy link
Collaborator

@litlfred litlfred Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot apply changes based on this feedback

Copy link
Contributor Author

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes already applied in commit 80d0873. The regex pattern is now defined as the module-level constant VALUESET_PATTERN and used in both parsing functions.

input/scripts/update_sushi_config.py Outdated
model_id = logical_match.group(1)

# Extract title
title_match = re.search(r'^Title:\s*"([^"]+)"', content, re.MULTILINE)
Copy link

Copilot AI Oct 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The regex patterns for Title and Description are duplicated between both parsing functions. Consider extracting these into shared constants: TITLE_PATTERN = r'^Title:\s*\"([^\"]+)\"' and DESCRIPTION_PATTERN = r'^Description:\s*\"([^\"]+)\"'

Copilot uses AI. Check for mistakes.
Copy link
Collaborator

@litlfred litlfred Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot apply changes based on this feedback

Copy link
Contributor Author

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes already applied in commit 80d0873. The Title and Description patterns are now defined as module-level constants TITLE_PATTERN and DESCRIPTION_PATTERN, shared between both parsing functions.

input/scripts/update_sushi_config.py Outdated
title = title_match.group(1) if title_match else model_id

# Extract description
desc_match = re.search(r'^Description:\s*"([^"]+)"', content, re.MULTILINE)
Copy link

Copilot AI Oct 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The regex patterns for Title and Description are duplicated between both parsing functions. Consider extracting these into shared constants: TITLE_PATTERN = r'^Title:\s*\"([^\"]+)\"' and DESCRIPTION_PATTERN = r'^Description:\s*\"([^\"]+)\"'

Copilot uses AI. Check for mistakes.
Copy link
Collaborator

@litlfred litlfred Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot apply changes based on this feedback

Copy link
Contributor Author

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes already applied in commit 80d0873. The Title and Description patterns are now defined as module-level constants TITLE_PATTERN and DESCRIPTION_PATTERN, shared between both parsing functions.

input/scripts/update_sushi_config.py Outdated
valueset_id = valueset_match.group(1)

# Extract title
title_match = re.search(r'^Title:\s*"([^"]+)"', content, re.MULTILINE)
Copy link

Copilot AI Oct 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The regex patterns for Title and Description are duplicated between both parsing functions. Consider extracting these into shared constants: TITLE_PATTERN = r'^Title:\s*\"([^\"]+)\"' and DESCRIPTION_PATTERN = r'^Description:\s*\"([^\"]+)\"'

Copilot uses AI. Check for mistakes.
Copy link
Collaborator

@litlfred litlfred Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot apply changes based on this feedback

Copy link
Contributor Author

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes already applied in commit 80d0873. The Title and Description patterns are now defined as module-level constants TITLE_PATTERN and DESCRIPTION_PATTERN, shared between both parsing functions.

input/scripts/update_sushi_config.py Outdated
title = title_match.group(1) if title_match else valueset_id

# Extract description
desc_match = re.search(r'^Description:\s*"([^"]+)"', content, re.MULTILINE)
Copy link

Copilot AI Oct 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The regex patterns for Title and Description are duplicated between both parsing functions. Consider extracting these into shared constants: TITLE_PATTERN = r'^Title:\s*\"([^\"]+)\"' and DESCRIPTION_PATTERN = r'^Description:\s*\"([^\"]+)\"'

Copilot uses AI. Check for mistakes.
Copy link
Collaborator

@litlfred litlfred Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot apply changes based on this feedback

Copy link
Contributor Author

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes already applied in commit 80d0873. The Title and Description patterns are now defined as module-level constants TITLE_PATTERN and DESCRIPTION_PATTERN, shared between both parsing functions.

@litlfred
Refactor: Extract FSH regex patterns into module-level constants
80d0873 
Co-authored-by: litlfred <662242+litlfred@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@litlfred litlfred Awaiting requested review from litlfred

Assignees

@litlfred litlfred

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

LM and valusets not found

2 participants

@litlfred