Create VCU Simulation Framework for Integration Testing
kevinl03 opened this issue · comments
Issue Summary:
Develop a comprehensive VCU Simulation Framework that empowers users to conduct detailed testing of the vehicle's firmware under race-like scenarios. This framework will facilitate thorough verification of the VCU's behavior in response to various critical inputs and stimuli, including
-APPS (Accelerator Pedal Position Sensor)
-BSE (Brake System Enable) pedals
-RTD (Ready-To-Drive) signals
-Set-Reset-Flip Events
-TSAL (Transmission Shift Actuator Lock).
Scope and Purpose:
The primary goal of this framework is to enable rigorous integration testing of the VCU's behavior. It will provide a controlled environment where the VCU's responses to various inputs can be tested comprehensively. The testing scenarios will simulate real-world conditions, ensuring that the VCU functions correctly under different operational states.
Key Features:
-
Realistic Race-Like Scenarios: The framework will allow users to mimic race-like conditions, creating a dynamic testing environment that closely resembles actual vehicle operation.
-
Multi-Input Simulation: Users can simulate various critical inputs, such as accelerator and brake pedal actions, RTD signals, and shift actuator behaviors, to assess how the VCU responds.
-
Detailed Event and State Testing: The framework will offer in-depth testing of VCU events and states, ensuring that the VCU behaves as expected in every operational phase.
Benefits:
-
Thorough Verification: Integration testing with this framework will enable comprehensive verification of the VCU's functionality and ensure that it responds correctly to a wide range of inputs and stimuli.
-
Early Issue Detection: The ability to simulate real-world scenarios will allow for early detection of potential issues and improve the reliability of the VCU.
-
Robust Validation: The framework will validate the correctness of the VCU's events and states, enhancing the overall safety and performance of the vehicle.
Additional Information:
This issue seeks to create a flexible and effective VCU Simulation Framework that enhances integration testing. Your contributions and ideas are highly valued to ensure the successful implementation of this framework. Please provide your insights and suggestions on how to best achieve this objective.
The current simulation framework can address all of the features discussed above except realistic race-like scenarios. This is because we are using a non-deterministic machine (our laptop) to test a highly deterministic system (the VCU/BMS). Other drawbacks were acknowledged in the initial proposal here
These drawbacks are okay because we can still test the functionality and logic of our system. We should recognize this and avoid building tests with this framework that attempt to simulate fine-tune timing scenarios (as it will not be 100% accurate anyway)
The end goal would be to get back to the Test board project and apply this new test framework to integrate with it. From a high level, this would mean designing a new PCB with a microcontroller where values and expected results are sent to it from the testing framework. The PCB is then in charge of simulating and verifying real hardware signals in real time
This is one direction we could go (Hardware in the loop):
Benefits: Closer to real simulation
Drawbacks: Still just as inaccessible and slow in terms of development times as the current test framework ie we need real hardware, need to be in the lab, and can't run through CI/CD pipeline in the future
The other direction we could go is Software in the loop:
Benefits: Can run without hardware making it easy for a lot of people to develop, can easily create a CI/CD pipeline to run tests on every PR
Drawbacks: Further from reality (by a negligible amount?)
FreeRTOS has a windows and linux simulation that uses threads to mimic the RTOS runtime environment. Essentially, you can build and run your program on your laptop to test software logic and thread synchronization.
This could also use the existing testing framework but use external threads to simulate the input/output data. Personally, this is is the coolest fucking thing ever because you can easily onboard new members and have them run a complete environment testing and tinkering around with ~70% of the logic without ever needing the hardware. Also, you could run it in a container, and build and run tests on github on every PR, on every commit, or nightly
Compared to the current simulation, this could more finely test I/O logic.
For example, here's a (vague) requirement:
VCU must output shutdown signal at "correct times"
Currently we can verify this by checking that the VCU enters the fault state at the correct times and then assuming the fault state correctly outputs the shutdown signal
This SIL framework can directly check that the VCU outputs a (fake) shutdown signal at the correct time.
When I initially proposed the current testing framework, the VCU/BMS had no proper testing on it whatsoever so I tried to leverage the best of both Hardware in the loop (HIL) and Software in the loop (SIL) frameworks. It's a bit of a hodge podge where it is...
SIL (because):
Simulates fake values with Python
Doesn't require additional hardware
Tests software logic
HIL (because):
Runs on actual hardware
Does (kinda?) use real interrupts and logging through UART
Now we need to decide whether we want to lean into the SIL or HIL side. I've worked at companies with both SIL/HIL testing, and others with just HIL. Ideally, we would have every possible type of testing, but we need to consider what gives us the most benefit at the moment (considering new members joining, state of firmware, % of hardware vs software logic, rules and requirements, etc)
I think we should first try SIL since it has no other team dependencies (like Power for PCB design) and because as of now, we're still working on making stable versions of the VCU/BMS. Once we have those in beta testing, we can begin thinking about HIL. For new members and even current members on the team, SIL would also be a great resource to anyone who needs to get familiar with the VCU/BMS (like you mentioned with onboarding) but also with other sub-teams that may want to cross-check their work.
I think we should first try SIL since it has no other team dependencies (like Power for PCB design) and because as of now, we're still working on making stable versions of the VCU/BMS. Once we have those in beta testing, we can begin thinking about HIL. For new members and even current members on the team, SIL would also be a great resource to anyone who needs to get familiar with the VCU/BMS (like you mentioned with onboarding) but also with other sub-teams that may want to cross-check their work.
100% agree! I think I was very clearly biased towards SIL when I made these comments lol
Personally, this is is the coolest fucking thing ever because you can easily onboard new members and have them run a complete environment testing and tinkering around with ~70% of the logic without ever needing the hardware.
Interesting that the real-time scenario drawback was not fully articulated prior the development of the framework. Nevertheless, many parts can still be used internally within Team Phantom such as Google Cloud storage, Data Generation, and the entire command-line interface if we plan to intergrate similar sandbox testing for the BMS.
But since Software In the Loop testing seems like the favourite, the next question would be how would we integrate the tests/simulations/waves into the new SIL testing infrastructure? The addition of CI/CD within the VCU repositories is a great idea for development, and it would be awesome if the test generation, involving the linear transofrmation of voltages of the APPS and BSE sensors, can be intergrated within the CI/CD tools.
For example, an automated test to check for APPS1 and APPS2 discontinuity would follow similar form to this
The easy part is having unit tests to validate FSAE rules straight from the rulebook #59. The more complicated part would be integrating the testing framework models through an automated pipeline such as CI/CD
This would most likely mean we would need to have a set of race like-scenerious using specific models, as well as an expected output for each scenerio. These would act as integration tests/stress tests, and would be tested for each Git Push or something similar
On the other hand, we can simply leave this test framework out of the scope of automated validation, and instead keep it as a sandbox using the RTOS tools mentioned above. The main factors we would need to consider are the requirements of the this RTOS sandbox, and how we will set up an environment to simulate the VCU in real-life race scenarios. Eventually we would want this sandbox to also be compatible with testing the VCU when other vehicle components are in play.
@rafguevara14, I think it could be useful to define the next testing milestones and defining prioties
We already started on the Unit Testing #59 which we can intergrate within CI/CD once that is complete.
For the FreeRTOS environment, I would like to get your insight on this, especially with how we plan to migrate the current framework into an environment which will mimic RTOS.
Lastly, @kkramer215, this framework ultimately was designed to have freedom for generating tests for the BMS. If you believe the current architecture may have some drawbacks for the BMS testing, this could also be discussed.