SRIOV - sriov_numvfs is not persisted properly after reboot

[xagent003]

Blocking issue: The VFs need to be persisted before hostagent is started. Or specifically, before pf9-neutron-sriov-agent is started. Otherwise things break after reboot.

THings were working fine, then it was observed that hosts were rebooted. When they came up, hostagent was first started, which starts sriov-agent. At this time, no VFs were configured on the NIC. sriov-agent initializes some runtime state, and detected that NIC as having no VFs. Afterwards, per /var/log/messages, I saw the script /pf9-virtual-functions.sh being invoked and setting /sys/class/net/$1/device/sriov_numvfs

So those VFs were not monitored for changes later, when they were configured to VMs and neutron ports.

This can be fixed in 3 ways:

1. Properly persisting the numvfs in /etc/network/interfaces (Ubuntu) or in ifup scripts for the NIC (RedHat/Centos). This is how the official Openstack docs say: https://docs.openstack.org/neutron/rocky/admin/config-sriov.html

2. We can restart pf9-neutron-sriov-agent again in the script after setting numvfs parameter

3. Change systemd service dependencies so pf9-sriov-vf-manager.service is started BEFORE pf9-hostagent.

Since this isn't committed, this needs to be changed manually whereever for any customers already using this branch locally. I don't have access to their machines or know where.