v560tu/v540tu seems slow on IO, needs confirmation

[tlaurion]

Only thing I can do is compare across devices.

This is systemd-analyze blame output, for reference

My nv41 has bigger templates and is my main driver.

nv41 64 GB ram, drive: Samsung SSD 980 PRO 2TB

v560tu 64gb ram, drive: SSDPR-PX700-04T-80

cc @macpijan

[macpijan]

I assume it can be useful to document disk/memory in both setups as well

[tlaurion]

I assume it can be useful to document disk/memory in both setups as well

Modified OP @macpijan

[tlaurion]

Not sure. Reencryption works @1.4GiB/s, with power draw of 35w, no fan spinning.

[mkopec]

What are the models of SSDs in both laptops? V560TU could have a DRAM-less disk which are much slower in random I/O

[macpijan]

I think it's reasonably to assume disks are from what @wessel-novacustom is offering in the configurator.

V56 offers only one option: https://novacustom.com/product/v56-series/

While NV41 used to offer Samsung disks: https://novacustom.com/product/nv41-series/

Can we please summarize the exact hw configurations being compared here?

[mkopec]

NV41 can come with 980 Pro (with DRAM cache) or 980 (DRAM-less). Apparently the PX700 disk in V560TU does not have DRAM cache but goodram sells it as HMB 3.0 which uses host RAM for caching. I have no idea whether or not that works in Qubes

[wessel-novacustom]

@macpijan @mkopec

I'm not sure if that's actually the issue.

If I compare I/O write and especially read operations (especially with small files), the -TU models are performing bad compared to the NVIDIA variants. Now that's for UEFI of course, but might still make sense to compare to see if there is a difference and why.

[mkopec]

Here's what it looks like with V560TU on my side (96GB / 2TB)

[tlaurion]

@mkopec @macpijan @wessel-novacustom : updated OP

nv41 64 GB ram, drive: Samsung SSD 980 PRO 2TB

and

v560tu 96gb ram, drive: SSDPR-PX700-04T-80

Opened issue QubesOS/qubes-issues#9723

Want me to run some tests? Please detail.

Also note that as pointed out at #1889 (comment):

Dasharo/coreboot@94e5f5d...048ca83

When we attempted that coreboot version bump, perf worsen to the point of systemd services timing out on first stage of qubesos installation, and templates installation took more than an hour.

[tlaurion]

Also note that as pointed out at #1889 (comment):

Dasharo/coreboot@94e5f5d...048ca83

When we attempted that coreboot version bump, perf worsen to the point of systemd services timing out on first stage of qubesos installation, and templates installation took more than an hour.

Some additional notes from testing, will edit (re-testing https://github.com/tlaurion/heads/tree/perf_comparison_with_reverted_coreboot_version_bump_v560tu)

• ram init takes around 2m on 64gb ram, not 1
• Heads fb refresh (fbwhiptail drawing) slower.
• Heads hashing of /boot content slower
• Resealing TPM Disk Unlock Key (cpu based derivation of DRK to validate slot + new key from luks DRK into new slot) way slower

Insight is not only IO being slower, but as if CPU speed was limited to be turtle speed as well.

systemd-analyze blame with branch including changes to coreboot version bump:

full log:

systemd-analyze_blame.txt

CC @macpijan @mkopec

[tlaurion]

systemd-analyze blame with 36e30d0 for comparison.

Exerpt:

32.838s qubes-vm@sys-whonix.service
26.821s qubes-vm@sys-firewall.service
26.016s dev-disk-by\x2duuid-9254e32c\x2df3a1\x2d4ddf\x2d917d\x2d1496ed47d5d6.device
26.016s dev-disk-by\x2dpartuuid-3f923856\x2dba50\x2d4c86\x2d9d79\x2dcb60c453af91.device
26.016s dev-nvme0n1p3.device
26.016s sys-devices-pci0000:00-0000:00:06.0-0000:01:00.0-nvme-nvme0-nvme0n1-nvme0n1p3.device
26.016s dev-disk-by\x2did-nvme\x2dnvme.1e4b\x2d473441303037343132\x2d53534450522d50583730302d3034542d3830\x2d00000001\x2dpart3.device
26.016s dev-disk-by\x2dpath-pci\x2d0000:01:00.0\x2dnvme\x2d1\x2dpart3.device
26.016s dev-disk-by\x2did-nvme\x2dSSDPR\x2dPX700\x2d04T\x2d80_G4A007412\x2dpart3.device
25.993s dev-disk-by\x2dpartuuid-3a30b820\x2d3a74\x2d41fb\x2d938d\x2d5e0288966fd4.device
25.993s sys-devices-pci0000:00-0000:00:06.0-0000:01:00.0-nvme-nvme0-nvme0n1-nvme0n1p2.device
25.993s dev-nvme0n1p2.device
25.993s dev-disk-by\x2did-nvme\x2dnvme.1e4b\x2d473441303037343132\x2d53534450522d50583730302d3034542d3830\x2d00000001\x2dpart2.device
25.993s dev-disk-by\x2did-nvme\x2dSSDPR\x2dPX700\x2d04T\x2d80_G4A007412\x2dpart2.device
25.993s dev-disk-by\x2duuid-10f43b72\x2d51a0\x2d40ce\x2d8407\x2dc31d8e29b67e.device
25.993s dev-disk-by\x2dpath-pci\x2d0000:01:00.0\x2dnvme\x2d1\x2dpart2.device
25.980s dev-disk-by\x2dpath-pci\x2d0000:01:00.0\x2dnvme\x2d1\x2dpart4.device
25.980s dev-disk-by\x2duuid-fb504cfb\x2d542a\x2d4b62\x2da176\x2d515781072d00.device
25.980s dev-nvme0n1p4.device
25.980s sys-devices-pci0000:00-0000:00:06.0-0000:01:00.0-nvme-nvme0-nvme0n1-nvme0n1p4.device
25.980s dev-disk-by\x2did-nvme\x2dSSDPR\x2dPX700\x2d04T\x2d80_G4A007412\x2dpart4.device
25.980s dev-disk-by\x2dpartuuid-6675f426\x2d7f58\x2d47ec\x2dba1a\x2d54a321a95c54.device
25.980s dev-disk-by\x2did-nvme\x2dnvme.1e4b\x2d473441303037343132\x2d53534450522d50583730302d3034542d3830\x2d00000001\x2dpart4.device
25.832s dev-disk-by\x2did-nvme\x2dSSDPR\x2dPX700\x2d04T\x2d80_G4A007412\x2dpart1.device
25.832s dev-nvme0n1p1.device
25.832s dev-disk-by\x2dpartuuid-bf182adf\x2da263\x2d4c81\x2da12a\x2d789f69fb6263.device
25.832s sys-devices-pci0000:00-0000:00:06.0-0000:01:00.0-nvme-nvme0-nvme0n1-nvme0n1p1.device
25.832s dev-disk-by\x2did-nvme\x2dnvme.1e4b\x2d473441303037343132\x2d53534450522d50583730302d3034542d3830\x2d00000001\x2dpart1.device
25.832s dev-disk-by\x2dpath-pci\x2d0000:01:00.0\x2dnvme\x2d1\x2dpart1.device
25.745s dev-ttyS4.device
25.745s sys-devices-pci0000:00-0000:00:1e.0-dw\x2dapb\x2duart.3-dw\x2dapb\x2duart.3:0-dw\x2dapb\x2duart.3:0.0-tty-ttyS4.device
25.738s dev-ttyS19.device
25.738s sys-devices-platform-serial8250-serial8250:0-serial8250:0.19-tty-ttyS19.device
25.737s dev-ttyS16.device
25.737s sys-devices-platform-serial8250-serial8250:0-serial8250:0.16-tty-ttyS16.device
25.736s sys-devices-platform-serial8250-serial8250:0-serial8250:0.1-tty-ttyS1.device
25.736s dev-ttyS1.device
25.735s sys-devices-platform-serial8250-serial8250:0-serial8250:0.13-tty-ttyS13.device
25.735s dev-ttyS13.device
25.734s dev-ttyS10.device
25.734s sys-devices-platform-serial8250-serial8250:0-serial8250:0.10-tty-ttyS10.device
25.733s sys-devices-platform-serial8250-serial8250:0-serial8250:0.18-tty-ttyS18.device
25.733s dev-ttyS18.device
25.733s sys-devices-platform-serial8250-serial8250:0-serial8250:0.15-tty-ttyS15.device
25.733s dev-ttyS15.device
25.729s sys-devices-platform-serial8250-serial8250:0-serial8250:0.0-tty-ttyS0.device
25.729s dev-ttyS0.device
25.729s sys-devices-platform-serial8250-serial8250:0-serial8250:0.20-tty-ttyS20.device
25.729s dev-ttyS20.device

So similar (but still slower) compared to @mkopec dump at #1894 (comment):

I guess some discards/trim operations changed the numbers. Will reinstall with qubes defaults and LVM and dump last numbers on my side on next comment.

systemd-analyze_blame_master.txt

CC @macpijan @mkopec

[macpijan]

If I compare I/O write and especially read operations (especially with small files), the -TU models are performing bad compared to the NVIDIA variants. Now that's for UEFI of course, but might still make sense to compare to see if there is a difference and why.

I agree this is to be checked. Did we have a tracking of this problem already?

Also, do we want to do it right now?
Should this be gating heads release if (to be confirmed) performance would be similar on the same device across UEFI and heads firmware?

Ideally, we release heads release from the same (similar) coreboot base as the previous UEFI release.

[macpijan]

Some additional notes from testing, will edit (re-testing https://github.com/tlaurion/heads/tree/perf_comparison_with_reverted_coreboot_version_bump_v560tu)

* ram init takes around 2m on 96gb ram, not 1

* Heads fb refresh (fbwhiptail drawing) slower.

* Heads hashing of /boot content slower

* Resealing TPM Disk Unlock Key (cpu based derivation of DRK to validate slot + new key from luks DRK into new slot) way slower

Insight is not only IO being slower, but as if CPU speed was limited to be turtle speed as well.

systemd-analyze blame with branch including changes to coreboot version bump:

Thanks for this test, this will be useful testing point for the future. We should focus on testing what we are aiming to release right now, however.

We propose that we both confirm once again that the binary from this commit is "OK" in terms of performance as reported by us here previously: #1894 (comment)

Where "OK" means "comparable to the existing UEFI release in the same hardware specification", not "strictly better than previous laptop model in this specific benchmark", especially that devices with different hardware configurations are being compared here.

If confirmed, we propose that we use this commit as a release for Dasharo+heads @tlaurion @wessel-novacustom to not postpone it any longer.

We can continue investigating the performance concerns, such as raised by @wessel-novacustom here #1894 (comment) in individual dasharo issues.

[wessel-novacustom]

Ideally, we release heads release from the same (similar) coreboot base as the previous UEFI release.

• I can see that, but this performance issue is one of the biggest issues the -TU series has.

Ideally, it would be fixed, despite the coreboot base being slightly different in that case.

In case a coreboot patch could be made to fix this issue, I'm interested in getting a link to that patch, so I can assist customers with UEFI firmware who are complaining about this.

My suggestion is to make a v0.9.2-rc1/v1.0.0-rc1 untested UEFI release with just this fix patch. We can then use that version as the base for Heads.