This repository hosts the various test cases offered out-of-the-box by 5GASP's CI/CD Stack. The test cases can be found in fpt/tests. Each test case is defined through a Robot script, aided by a requirements.txt file that lists the various test case dependencies.
This repository offers a vast collection of tests than may be used (i) through 5GASP's CI/CD Pipeline and (ii) locally. Before onboarding your application in 5GASP's CI/CD Pipeline, you should first perform these tests manually.
To simplify the local testing processes, this repository also makes available 4 bash scripts that execute the different tests:
/ftp/tests/run_5G_readiness_tests.sh/ftp/tests/run_performance_and_scalability_tests.sh/ftp/tests/run_security_and_privacy_tests.sh/ftp/tests/run_availability_and_continuity_tests.sh(Note: The Availability and Continuity tests can only be performed at UoP's testbed)
Before running these scripts, you shoul first update the configuration variables of each test. When you have look at the scripts, you will see that updating the configuration variables is a straightforward process.
After updating the configuration variables, you may execute the different scripts.
The scripts will create a results directory, where the results will be stored. Furthermore, when the script finishes running all tests, you will also be presented with a list of the tests that failed. You may then get into the results directory and find the log and report files for these failed tests. Those files shall help you figure out the reason of the tests' failure.
To deploy the LTR, you must first update the docker-compose.yaml to reflect the IP on which the LTR will be exposed. You may also update the FTP server credentials, but that is not mandatory. After updating the exposure IP, you may run docker compose up, and the LTR will become available.
As the 5GASP LTR is the host of 5GASP's pre-defined test cases, it will be queried by the 5GASP Testing Agents to obtain the test cases that participate in the Network Application testing processes. Nevertheless, to achieve this, it is needed to integrate both components. This can be achieved by configuring global variables and credentials in the Testing Agents employed for Network Application validation. Therefore, you should defined the following secrets (Global Store / Plain Credentials) in your Testing Agent (offered through Jenkins):
ltr_user: should reflect the user configured in thedocker-compose.yamlltr_password: should reflect the password configured in thedocker-compose.yamlltr_location: should reflect the location of the LTR server
The testing of 5GASP's Network Applications, relies on 3 main components:
- 2 instances of the MiniAPI (1 shall be used as a server and the other to mimic a UE)
- NEF Emulator Instance
Both the NEF Emulator and one of the MiniAPI instances can be manually deployed before the testing process starts. This means that at least, one instance of the MiniAPI should be comprised by your application. As such, your Network Application is responsible for providing a MiniAPI instance.
The need for having 2 instances of the MiniAPI relies on the requirements of several test cases, mostly performance related ones. These tests rely, for instance, on iPerf3. Therefore, for these tests, it is required to have 1 iPerf3 server and 1 iPerf3 client. The iPerf3 server can be offered by one of the MiniAPI instances, while the iPerf3 client should be provided by the other MiniAPI instance.
In the case of CNFs, the MiniAPI comprised by the Network Application shall be used as a client, and the manually deployed one should be used as a server. This is due to the fact that, to expose an iPerf3 server in a CNF, one would have to update the K8s Services to expose this component. To make things easier, it is proposed that the manually deployed MiniAPI is then used as a server and the one encapsulated by the application used as a server.
For VNFs, the MiniAPI comprised by the Network Application shall be used as a server, and the manually deployed one will be the client.
In regards to the OS dependencies required to run the MiniAPI, you may find them at: https://github.com/5gasp/NetworkAppControl-MiniAPI. This repository also includes instructions on how to deploy your application's MiniAPI.
When adding the MiniAPI to your application, remember that the MiniAPI should be automatically started alongside all other Network Applicaiton components. The configuration needed to achieve this is completely delegated to the Network Application owners.
Due to the intrincacies of some tests, the MiniAPI placement is highly important. Def14Perf11 and Def14Perf7 test cases demand for the MiniAPI to be placed in the some component that offers the APIs that shall be tested.
The first test case (Def14Perf11) involves testing one of the Network Application offered APIs in regards to the maximum number of simultaneous connections it can handle. Therefore, the the tested API and the MiniAPI should be located inside the same VNF/CNF, to allow the MiniAPI to monitor the number of simultaneous connections.
The second test case (Def14Perf7) is very similar to the first. However, it targets a specific API - the one that will handle the NEF QoS and Location callbacks. Once again, this API and the MiniAPI should be located inside the same VNF/CNF.
Please take this into consideration when providing your Network Application MiniAPI(s) - a Network Application can provide several MiniAPIs.
To illustrate the abovementioned scenario, you may have a look at the following diagram.
This diagram assumes that both the NEF callback target API and the target API for the Def14Perf11 Test are comprised by VNF1.
An alternative scenario would be the following one.
In this scenario, the aforementioned APIs can be located either inside VNF1 or inside VNF2.