[BUG] Backend Pools cannot be created when they depend on individual Backends

[dan-korczak]

Release version

APIOps Toolkit for Azure APIM v6.0.1.10

Describe the bug

When both Backend Pools and Backends are defined in configuration, if a Backend Pool targets a Backend, it cannot be created.

There is no ordering or dependency management where Backends forced to be created before Backend Pools.

info: publisher[0]
      Putting backends...
info: publisher[0]
      Putting backend backend-pool-secondary...
info: publisher[0]
      Putting backend backend-west...
crit: publisher[0]
      Application failed.
      System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/.../resourceGroups/.../providers/Microsoft.ApiManagement/service/.../backends/backend-pool-secondary?api-version=2023-09-01-preview failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"One or more fields contain incorrect values:","details":[{"code":"ValidationError","target":"pool","message":"Backend Entity 'backend-west' not found."}]}}'.
         at common.HttpPipelineExtensions.<>c__DisplayClass11_0.<PutContent>b__0(Response response) in /home/runner/work/apiops/apiops/tools/code/common/Http.cs:line 173
         at LanguageExt.Either.Left`2.Match[B](Func`2 Left, Func`2 Right)
         at LanguageExt.Either`2.IfLeft(Func`2 leftMap)
         at common.HttpPipelineExtensions.PutContent(HttpPipeline pipeline, Uri uri, BinaryData content, CancellationToken cancellationToken) in /home/runner/work/apiops/apiops/tools/code/common/Http.cs:line 173
         at common.BackendModule.PutDto(BackendUri uri, BackendDto dto, HttpPipeline pipeline, CancellationToken cancellationToken) in /home/runner/work/apiops/apiops/tools/code/common/Backend.cs:line 415
         at publisher.BackendModule.<>c__DisplayClass11_0.<<GetPutBackendInApim>b__0>d.MoveNext() in /home/runner/work/apiops/apiops/tools/code/publisher/Backend.cs:line 172
      --- End of stack trace from previous location ---
         at publisher.BackendModule.<>c__DisplayClass7_1.<<GetPutBackend>b__1>d.MoveNext() in /home/runner/work/apiops/apiops/tools/code/publisher/Backend.cs:line 121
      --- End of stack trace from previous location ---
         at common.OptionExtensions.IterTask[T](Option`1 option, Func`3 action, CancellationToken cancellationToken) in /home/runner/work/apiops/apiops/tools/code/common/Option.cs:line 22
         at common.OptionExtensions.IterTask[T](Option`1 option, Func`2 action) in /home/runner/work/apiops/apiops/tools/code/common/Option.cs:line 16
         at publisher.BackendModule.<>c__DisplayClass7_0.<<GetPutBackend>b__0>d.MoveNext() in /home/runner/work/apiops/apiops/tools/code/publisher/Backend.cs:line 121
      --- End of stack trace from previous location ---
         at System.Threading.Tasks.Parallel.<>c__53`1.<<ForEachAsync>b__53_0>d.MoveNext()
      --- End of stack trace from previous location ---
         at common.IEnumerableExtensions.IterParallel[T](IEnumerable`1 enumerable, Func`3 action, Int32 maxDegreeOfParallelism, CancellationToken cancellationToken) in /home/runner/work/apiops/apiops/tools/code/common/Enumerable.cs:line 61
         at common.IEnumerableExtensions.IterParallel[T](IEnumerable`1 enumerable, Func`3 action, CancellationToken cancellationToken) in /home/runner/work/apiops/apiops/tools/code/common/Enumerable.cs:line 47
         at publisher.BackendModule.<>c__DisplayClass1_0.<<GetPutBackends>b__0>d.MoveNext() in /home/runner/work/apiops/apiops/tools/code/publisher/Backend.cs:line 54
      --- End of stack trace from previous location ---
         at publisher.AppModule.<>c__DisplayClass1_0.<<GetRunApplication>b__0>d.MoveNext() in /home/runner/work/apiops/apiops/tools/code/publisher/App.cs:line 173
      --- End of stack trace from previous location ---
         at common.HostingModule.RunApplication(IHost host) in /home/runner/work/apiops/apiops/tools/code/common/Hosting.cs:line 89
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...

Expected behavior

In this example:

1. Backend backend-west should be created first
2. Backend Pool backend-pool-secondary should only then be created

Therefore, a Backend Pool can be created pointing at one or more backends that were also created.

Actual behavior

No ordering is forced. In some executions, Backend creation is executed first, and then the Backend Pool. In other executions, the Backend pool is executed first without the Backend having been created (crashing the workflow).

This issue compounds when a Backend Pool specifies larger and larger quantifies of Backends.

Reproduction Steps

configuration.env.yml

...

backends:
  - name: backend-east
    properties:
      url: &backend-east-url https://xyz.123.eastus.azurecontainerapps.io
  - name: backend-west
    properties:
      url: &backend-west-url https://xyz.123.westus.azurecontainerapps.io
  - name: backend-pool-primary
    properties:
      pool:
        services:
          - id: /subscriptions/.../resourceGroups/.../providers/Microsoft.ApiManagement/service/.../backends/backend-east
            priority: 1
          - id: /subscriptions/.../resourceGroups/.../providers/Microsoft.ApiManagement/service/.../backends/backend-west
            priority: 2
  - name: backend-pool-secondary
    properties:
      pool:
        services:
          - id: /subscriptions/.../resourceGroups/.../providers/Microsoft.ApiManagement/service/.../backends/backend-east
            priority: 1
          - id: /subscriptions/.../resourceGroups/.../providers/Microsoft.ApiManagement/service/.../backends/backend-west
            priority: 2


...

[github-actions]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

[shaunaa126]

We are also receiving this issue and is a major blocker for us. Maybe a small enhancement to include BackendPools into a separate directory so that the publisher can deploy backends and backendpool in correct order.

[guythetechie]

Should be fixed in v7. Here is the current alpha release.