Skip to content

Conversation

@vanessavmac
Copy link
Collaborator

@vanessavmac vanessavmac commented Oct 18, 2025

Summary

Follow up to #910 which implemented the controller logic for managing batch image processing tasks for ML Jobs.

This PR focuses on the "consumer" side of the batch image processing system by demonstrating how a celery worker can be added to a processing service.

List of Changes

  • Adds a celery worker to each template processing service to demonstrate how the async distributed ML backend can be utilized by external celery workers.

Related Issues

Addresses in part #515

How to Test the Changes

Apply the same testing as #910. Inside of flower, you can see that process_pipeline_request tasks are picked up by both the antenna and the ML backend celery worker.

Checklist

  • I have tested these changes appropriately.
  • I have added and/or modified relevant tests.
  • I updated relevant documentation or comments.
  • I have verified that this PR follows the project's coding standards.
  • Any dependent changes have already been merged to main.

@vanessavmac vanessavmac requested a review from mihow October 18, 2025 06:15
@netlify
Copy link

netlify bot commented Oct 18, 2025

Deploy Preview for antenna-preview ready!

Name Link
🔨 Latest commit 649a628
🔍 Latest deploy log https://app.netlify.com/projects/antenna-preview/deploys/68f3307dbcdafc00088c625c
😎 Deploy Preview https://deploy-preview-1011--antenna-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
Lighthouse
Lighthouse
1 paths audited
Performance: 31 (🔴 down 34 from production)
Accessibility: 80 (no change from production)
Best Practices: 100 (no change from production)
SEO: 92 (no change from production)
PWA: 80 (no change from production)
View the detailed breakdown and full score reports

To edit notification comments on pull requests, go to your Netlify project configuration.

@vanessavmac vanessavmac changed the title feat: add external celery workers to the processing services Enhance processing service templates to consume from pipeline queues Oct 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants