Skip to content

Conversation

@christopher-hakkaart
Copy link
Member

This PR adds documentation for Wave's bin script bundling behavior in Nextflow pipelines. This guide helps users understand which scripts get packaged into container images and how this affects container fingerprinting and caching.

This PR is intentionally limited in scope to address a specific user pain point: understanding how Wave handles bin scripts from different directories (${projectDir}/bin/ vs. ${moduleDir}/resources/usr/bin/) and when those scripts are bundled into container layers.

The guide covers:

  • Script location and Wave's bundling behavior for each location
  • Conditions under which project bin scripts are bundled
  • How script bundling affects Wave container fingerprinting
  • How to inspect containers to verify bundled scripts

I've tried to keep the scope limited and focus on the bin behavior without going too deep. Not covered here:

  • Broader documentation covering Wave configuration, container provisioning, build caching, and integration patterns with Nextflow workflows
  • Template and lib files/scripts
  • End-to-end documentation of Wave's container building, layer generation, registry operations, and caching mechanisms

Copy link
Member

@gwright99 gwright99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left comments and thoughts. Good first pass but I feel that elaboration could be helpful for the average reader who doesn't yet have a full grasp on the inner workings of these various ecosystem components.

@christopher-hakkaart christopher-hakkaart marked this pull request as ready for review December 1, 2025 02:37
Copy link
Member

@gwright99 gwright99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving to get PR unblocked and on understanding discussed items will be handled in separate document at some future point.

@christopher-hakkaart
Copy link
Member Author

@robsyme - I've added some of your content from #945

When you have a moment, could you review this PR and ensure it addresses your needs?


When you enable Fusion or use the AWS Fargate executor, Wave bundles scripts from the project `bin/` directory into a container layer. This provides better performance but ties scripts to specific execution environments. Scripts are added to `/usr/local/bin/` in the container image, making them directly executable without path qualification. Remote `bin/` directory upload is disabled.

Modifying scripts in the project `bin/` directory modify the container fingerprint for all Wave containers in the workflow. See [Wave container fingerprinting](#wave-container-fingerprinting) for more information.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit too technical, i'd focus it will result in creating a modified container, and therefore invalidating the cache for previous runs

Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@christopher-hakkaart christopher-hakkaart merged commit 21e78e1 into master Dec 14, 2025
1 check passed
@christopher-hakkaart christopher-hakkaart deleted the docs-guide-bin-scripts branch December 14, 2025 20:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants