[WIP] Add logic to the producer to be able to requeue failed jobs #250
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR covers the second part from #242, having the producer a flag to requeue jobs from the buried queue.
Right now in the buried queue should be stay only temporary-errors and fatal errors, so temporary errors are requeued and fatal errors are left on the buried queue.
Note that those jobs in the buried queue stayed as pending in the database, so there's no need to change their status when they are requeed.
There are two points to think about:
The--republish-jobsflag works as an addition to the producer regular job, so if we republish jobs from the buried queue and we give the producer a list of repositories where there are repositories that they are in the buried queue too, we will have duplicated jobs on the queue. An option could be have a command just to republish jobs.UPDATE: after talk with @ajnavarro this functionality should work as a service that only do the requeue task.
What do you think @src-d/data-retrieval?
This PR depends on src-d/framework#40