Celery task naming

[aitor-perez-epfl]

I found out that naming celery tasks with more than one dot breaks the function route_task in the celery_config.py file:

File "/Users/hera/graphai/graphai/core/celery_utils/celery_config.py", line 8, in route_task
    queue, _ = name.split(".")
ValueError: too many values to unpack (expected 2)

I was trying to use task names like text.wikify.extract_keywords. We should either fix this function to accept this kind of names or define a different task naming convention which makes sense.

What do you think @RamtinYazdanian?

[RamtinYazdanian]

I propose changing that line to:
queue = name.split(".")[0]
And the naming convention can remain the same as it is now: queue.taskname. Does that sound good?

[aitor-perez-epfl]

That's fine for me. I was a bit worried however with task names not being meaningful enough, that's why I added the endpoint name. For instance, I have now one text.compute_scores task for concept detection (/text/wikify), but I can imagine having a similar one for another endpoint (e.g. /text/wikify_full) when I implement concept detection outside the ontology. How would you disambiguate those?

[RamtinYazdanian]

For the tasks where I have this kind of problem, I resolve it by including the "role" the task plays in the pipeline of tasks. For example, I have a task called "video.compute_noise_level_parallel" and another called "video.compute_noise_level_callback". Usually if there's no common element to extract from the two, I just add more underscored parts to the name.

…

On Tue, Apr 25, 2023, 1:50 PM Aitor Pérez ***@***.***> wrote: That's fine for me. I was a bit worried however with task names not being meaningful enough, that's why I asked the endpoint name. For instance, I have now one text.compute_scores task for concept detection (/text/wikify), but I can imagine having a similar one for another endpoint (e.g. /text/wikify_full) when I implement concept detection outside the ontology. How would you disambiguate those? — Reply to this email directly, view it on GitHub <#28 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEVE2A6S5D3Y7QQK5T7DR2LXC6263ANCNFSM6AAAAAAXK4AOQE> . You are receiving this because you were assigned.Message ID: ***@***.***>

[aitor-perez-epfl]

The role would be the same, in this case. They would both be callbacks. Imagine them as exchangeable pieces of the same workflow, only with some minor differences.

[RamtinYazdanian]

One idea is that since more than one dot is now allowed, you could disambiguate by adding the name of the endpoint it's for.

…

On Tue, Apr 25, 2023, 1:59 PM Aitor Pérez ***@***.***> wrote: The role would be the same, in this case. They would both be callbacks. Imagine them as exchangeable pieces of the same workflow, only with some minor differences. — Reply to this email directly, view it on GitHub <#28 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEVE2A43CFEHPOVERQOD5GTXC64DJANCNFSM6AAAAAAXK4AOQE> . You are receiving this because you were assigned.Message ID: ***@***.***>