fix: write provision.json on any node controller failures, add E2E

[awesomenix]

What type of PR is this?

/kind bug

What this PR does / why we need it:

aks node controller can have mutiple failures even before it starts running cse, capture everything

Which issue(s) this PR fixes:

Fixes #

Requirements:

• uses conventional commit messages
• includes documentation
• adds unit tests
• tested upgrade from previous version
• commits are GPG signed and Github marks them as verified

Special notes for your reviewer:

Release note:

none

[Devinwong]

LGTM. Thanks for improving the logging

[Copilot]

Copilot reviewed 47 out of 71 changed files in this pull request and generated 5 comments.

[Copilot]

Consider renaming ReturnErrorOnVMSSCreation to something more descriptive like AllowVMSSCreationFailure or PropagateVMSSCreationError, which better conveys its purpose of controlling error propagation behavior rather than just "returning" an error.

Suggested change

	// ReturnErrorOnVMSSCreation indicates whether to return error on VMSS creation failure or fail the test immediately.
	ReturnErrorOnVMSSCreation bool
	// PropagateVMSSCreationError controls whether VMSS creation failures should be propagated back to the caller as an error
	// (when true) or cause the test to fail immediately (when false).
	PropagateVMSSCreationError bool

[Copilot]

In the test Test_Ubuntu2204_Failure_Scriptless, there's a commented-out line that appears to be an alternative way to cause failure. This commented code should either be removed or have a clear comment explaining why it's kept for future reference.

Suggested change

//config.Version = "v200"

[Copilot]

In the panic recovery handler, writeCompleteFileOnError is being called inside the defer function. However, this function is also called at line 74 after runProvision returns. This could lead to duplicate file writes or unnecessary operations. Consider whether calling it within the defer is necessary, or if the panic should be allowed to propagate differently.

[Copilot]

In the writeCompleteFileOnError function, errors during JSON marshaling and directory/file creation are logged but not returned. While this is appropriate for a best-effort cleanup function, consider whether the function should return early after any failure to avoid attempting subsequent operations that may also fail.

Suggested change

	}
	baseDir := filepath.Dir(provisionJSONFilePath)
	if writeErr := os.MkdirAll(baseDir, 0755); writeErr != nil {
	slog.Error("failed to create directory for provision.json file", "path", baseDir, "error", writeErr)
	}
	if writeErr := os.WriteFile(provisionJSONFilePath, data, 0600); writeErr != nil {
	slog.Error("failed to write provision.json file", "path", provisionJSONFilePath, "error", writeErr)
	}
	} else {
	baseDir := filepath.Dir(provisionJSONFilePath)
	if writeErr := os.MkdirAll(baseDir, 0755); writeErr != nil {
	slog.Error("failed to create directory for provision.json file", "path", baseDir, "error", writeErr)
	} else if writeErr := os.WriteFile(provisionJSONFilePath, data, 0600); writeErr != nil {
	slog.Error("failed to write provision.json file", "path", provisionJSONFilePath, "error", writeErr)
	}
	}

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.