Symbol report verification report by pawelrutkaq · Pull Request #2544 · eclipse-score/score

pawelrutkaq · 2026-02-04T13:26:31Z

No description provided.

github-actions · 2026-02-04T13:26:43Z

⚠️ Docs-as-Code version mismatch detected
Please check the CI build logs for details and align the documentation version with the Bazel dependency.

github-actions · 2026-02-04T13:31:01Z

The created documentation from the pull request is available at: docu-html

docs/score_tools/symbol_report_blanket.rst

masc2023 · 2026-02-04T13:35:24Z

docs/score_tools/symbol_report_blanket.rst

+     - no
+     - yes
+     - no
+     - high


You did not cover bugs introduced by instrumentation, standard mitigation here would required running the test twice, once with and once w/o instrumentation

There is general problem: This is tool verification report and code coverage is a process. So how shall i describe it if i am specifically interested whether post procesing tooling require qualification. (here symbol report and blanket)

The code coverage think may part of another report or you must also explain it here as part of your use cases. And if the tools help to mitigate the rustc coverage issues, e.g. detect failure by instrumentation, than it should be added here

Reworked table

ok, now you removed that part of the process, may then good idea to create another report for the missing part of the process then already

aschemmel-tech

see inline comments, proposal: take over the failure modes and mitigations from https://public-docs.ferrocene.dev/main/certification/core/safety-plan/tools.html#code-coverage

aschemmel-tech · 2026-02-04T15:18:56Z

docs/score_tools/symbol_report_blanket.rst

+     - | The reported coverage number is reported higher than the real coverage.
+       | This may lead to false assumption that code is sufficiently covered by tests, although it is not.
+     - no
+     - Function will not be present in the report, so it will be visible that it is not covered.


Please describe better how this is visible, to be able to judge if it can be easily observed. E.g. if the coverage data is displayed in a view together with the source, I would belive that this is a high detection likelyhood. If it is only a seperate file logging all functions and their coverage I would see a low detection value appropriate.

Reworked table

aschemmel-tech · 2026-02-04T15:24:09Z

docs/score_tools/symbol_report_blanket.rst

+       | This may lead to false assumption that code is sufficiently covered by tests, although it is not.
+
+     - no
+     - Likelihood of this malfunction is low


No reason given for this statement. Where is the measure?

Reworked table

aschemmel-tech · 2026-02-04T15:24:59Z

docs/score_tools/symbol_report_blanket.rst

+     - | The reported coverage number is reported higher than the real coverage.
+       | This may lead to false assumption that code is sufficiently covered by tests, although it is not.
+
+     - no


safety impact must be yes - because testing is not complete.

Reworked table

aschemmel-tech · 2026-02-04T15:29:47Z

docs/score_tools/symbol_report_blanket.rst

+   * - 4
+     - Calculations are wrong, so the reported coverage number is not correct.
+     - | The reported coverage is different than the coverage in `profraw` files.
+     - no


A "real" coverage below 100% but calculated/displayed as 100% would be a safety problem.

Rounding can also become an issue.

Reworked table

aschemmel-tech · 2026-02-04T15:30:55Z

docs/score_tools/symbol_report_blanket.rst

+     - | The reported coverage is different than the coverage in `profraw` files.
+     - no
+     - no
+     - yes


What detection does this refer to?

Reworked table

aschemmel-tech · 2026-02-04T15:33:37Z

docs/score_tools/symbol_report_blanket.rst

+
+Result
+~~~~~~
+Symbol report and blanket do not require qualification for use in safety-related software development according to ISO 26262.


as stated by masc2023 - without additional measures I cannot agree to this result

Reworked table

docs/score_tools/symbol_report_blanket.rst

pahmann · 2026-02-04T16:31:50Z

docs/score_tools/symbol_report_blanket.rst

+Safety evaluation
+-----------------
+This section outlines the safety evaluation of symbol report and blanket for its use within the S-CORE project. This evaluation assumes that the Rust compiler is
+qualified and output of coverage data in `.profraw` format is correct. Due to that, we solely focus on post processing that is done by symbol report and blanket only.


Will be use and measure the coverage based on the .profraw somewhere else? This can have impact on the tool classification, which is the reason I am asking.

No, these tools are the only measurement for coverage of rust

docs/score_tools/symbol_report_blanket.rst

pahmann · 2026-02-04T16:34:13Z

docs/score_tools/symbol_report_blanket.rst

+   * - 4
+     - Calculations are wrong, so the reported coverage number is not correct.
+     - | The reported coverage is different than the coverage in `profraw` files.
+     - no


Rounding can also become an issue.

pahmann · 2026-02-04T16:42:49Z

docs/score_tools/symbol_report_blanket.rst

+   :widths: 1 2 8 2 6 4 2 2
+
+   * - Malfunction identification
+     - Use case description


Your use cases are actually malfunctions and not use cases. See https://eclipse-score.github.io/score/pr-2544/score_tools/doc_as_code.html#doc_tool__doc_as_code as an example of the use case and the resulting malfunction.

I reworded but kep same layout. I see the layout of table is different acros tools so...

Layout should be overall same, using https://eclipse-score.github.io/process_description//main/folder_templates/tools/tool_verification_report_template.html

pahmann · 2026-02-04T16:44:53Z

docs/score_tools/symbol_report_blanket.rst

+   :header-rows: 1
+   :widths: 1 2 8 2 6 4 2 2
+
+   * - Malfunction identification


Can there be anything as a case that:

The coverage is reported for another function.

There is a version mismatch that the tool takes the data from a previous or other run.

Could there be an issue for dynamic configuration like calling the unit test with configuration a but have configuration b mapped in code for the report?

I assume the use cases are limited which you want to do with the report, but the 4 malfunctions which you call use cases are incomplete.

added

The tools takes what is given by bazel as in picture. so its not tool reposiblity to handle that. But in any case of this, it would not casue reporting more coverage then it really is (realtivelly)

maybe, but that would make a report one big scrambled eggs ;)

pawelrutkaq · 2026-02-05T10:28:07Z

@pahmann @aschemmel-tech @masc2023 tried to resolve comments please recheck

PandaeDo · 2026-02-05T14:20:35Z

docs/score_tools/symbol_report_blanket.rst

+     - Overreporting, could result in testing gap.
+     - yes
+     - | Likelihood of such an error low due to wide usage of the tool (many S-CORE modules and other projects like ferrocene)
+       | Additionally, every new tool release is tested by running tests in prepared integration testsuite to detect such errors. (PROPOSAL POINT)


I would also add the expectation of the tester/user. So a failure can be detected.

PandaeDo · 2026-02-05T14:22:40Z

docs/score_tools/symbol_report_blanket.rst

+     - Underreporting, will not result in testing gap.
+     - yes
+     - Since we want to achieve 90%+ branch coverage this would stand out and be manually investigated.
+     - yes


Suggested change

- yes

- no

When I understand it correctly this will not have a impact on safety.

PandaeDo · 2026-02-05T14:23:45Z

docs/score_tools/symbol_report_blanket.rst

+     -  A function is not being considered, although it is part of the certified subset
+     - yes
+     - | `symbol-report` is developed to use exactly the same information as the compiler
+       | Additionally, every new tool release is tested by running tests in prepared integration testsuite to detect such errors. (PROPOSAL POINT)


Also in this case the user expectation will lead in some cases to a failure detection.

masc2023 · 2026-02-05T10:49:15Z

docs/score_tools/symbol_report_blanket.rst

+   :widths: 1 2 8 2 6 4 2 2
+
+   * - Malfunction identification
+     - Use case description


Layout should be overall same, using https://eclipse-score.github.io/process_description//main/folder_templates/tools/tool_verification_report_template.html

masc2023 · 2026-02-09T08:12:15Z

docs/score_tools/symbol_report_blanket.rst

+   :version: 1.90.0 (see [1])
+   :tcl: HIGH
+   :safety_affected: YES
+   :security_affected: NO


Why NO, security requires also coverage

masc2023 · 2026-02-09T08:13:25Z

docs/score_tools/symbol_report_blanket.rst

+----------------------------
+Installation
+~~~~~~~~~~~~
+| To add the Code coverage to your project or module follow guidelines in WIP


Is there a link to WIP available, or nothing at the moment, then add a issue to follow up with that and link it here

masc2023 · 2026-02-09T08:14:09Z

docs/score_tools/symbol_report_blanket.rst

+~~~~~~~~~~~
+Requires Rust toolchain and Bazel build environment.
+
+Safety evaluation


Compare template report and other examples, add missing Security Evaluation, you can keep it blank for now

masc2023 · 2026-02-09T08:15:57Z

docs/score_tools/symbol_report_blanket.rst

+     - Overreporting, could result in testing gap.
+     - yes
+     - | Likelihood of such an error low due to wide usage of the tool (many S-CORE modules and other projects like ferrocene)
+       | Additionally, every new tool release is tested by running tests in prepared integration testsuite to detect such errors. (PROPOSAL POINT)


What means Proposal Point? Additional measure to reduce the risk? Then add issue, link it here, that we can follow up as it then supports the argumentation why the rating is high
If Proposal Point is additional measure, update the table with additional measure needed yes

masc2023 · 2026-02-09T08:17:04Z

docs/score_tools/symbol_report_blanket.rst

+     - Underreporting, will not result in testing gap.
+     - yes
+     - Since we want to achieve 90%+ branch coverage this would stand out and be manually investigated.
+     - yes


are the 90% somewhere document, then link to it?

masc2023 · 2026-02-09T08:19:28Z

docs/score_tools/symbol_report_blanket.rst

+     - no
+     - yes
+     - no
+     - high


ok, now you removed that part of the process, may then good idea to create another report for the missing part of the process then already

Symbol report verification report

024a364

pawelrutkaq requested a review from a team as a code owner February 4, 2026 13:26

masc2023 requested changes Feb 4, 2026

View reviewed changes

pawelrutkaq added 3 commits February 4, 2026 14:36

Update doc-as-code

f5a7c8a

Add missing desc

970022e

Reword evaluation

d881e8f

aschemmel-tech requested changes Feb 4, 2026

View reviewed changes

pahmann requested changes Feb 4, 2026

View reviewed changes

Address review findings

567ba1e

pawelrutkaq requested review from aschemmel-tech, masc2023 and pahmann February 5, 2026 10:27

PandaeDo reviewed Feb 5, 2026

View reviewed changes

masc2023 requested changes Feb 9, 2026

View reviewed changes

Conversation

pawelrutkaq commented Feb 4, 2026

Uh oh!

github-actions bot commented Feb 4, 2026

Uh oh!

github-actions bot commented Feb 4, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pawelrutkaq Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aschemmel-tech left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pawelrutkaq commented Feb 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pawelrutkaq Feb 4, 2026 •

edited

Loading

aschemmel-tech left a comment •

edited

Loading