Testing

The testing of FhSim features is based on the Google C++ Testing Framework, see GTest Primer. FhSim provides a library with helper methods frequently used when testing features in FhSim and it's external libraries. They include code for parsing output files, calculating errors and standard setups of simulation tests. The methods compromise the library fhsim-testtools, which is used when testing SimObjects from external projects.

SimObject tests

A SimObject library developer can write tests for verification and validation of expected model behavior. This is typically done by producing expected output files, which are then compared against output from simulation runs. We recommend to put test implementation and data in a tests sub-directory in your project root.

A minimal setup structure for a SimObject library with a Pendulum SimObject is the following:

tests
├── CMakeLists.txt
├── in
│   └── Pendulum
│       ├── Pendulum_exp.txt
│       └── Pendulum_in.xml
├── Pendulum_Test.cpp
└── RunTests.cpp

• CMakeLists.txt is the build script for the tests folder.
• Pendulum_exp.txt is an expected output file from a simulation run. It should essentially be csv file that follows FhSim's output format.
• Pendulum_in.xml is the scenario file that describes the simulation run.
• Pendulum_Test.cpp contains the tests you want to run for a SimObject – in this case Pendulum.
• RunTests.cpp contains setup for creating the test executable, which runs the tests.

We provide excerpts of some of the mentioned files in the following.

CMakeLists.txt

add_executable(fhsim_example_test Pendulum_Test.cpp RunTests.cpp)
target_link_libraries(fhsim_example_test
    PRIVATE
        FhSim::fhsim
        FhSim::fhsim-testtools
        GTest::GTest)
add_test(
    NAME "fhsim_example"
    COMMAND fhsim_example_test "${CMAKE_CURRENT_SOURCE_DIR}" 0 4 n .
    WORKING_DIRECTORY ${PLAYPEN_DIR}/bin
    )

Pendulum_Test.cpp

#include "TestSetup.h"
 
namespace
{
const std::string dirname  = "Pendulum";
const double      simTime  = 2.0;
const double      realTime = 30.0;
} // namespace
 
TEST(SimObject, PendulumTest)
{
   testTools::SetUpTest(dirname, simTime, realTime, false);
   ASSERT_TRUE(initSim[0]); // Parsing of inputfile and initiation
   ASSERT_TRUE(initSim[1]); // Simulation
 
   std::vector<double> rmsError = testTools::RmsComparison(dirname);
   for (size_t i = 0; i < rmsError.size(); ++i) {
      EXPECT_NEAR(0, rmsError[i], 0.01);
   }
}

RunTests.cpp

#include "TestSetup.h"
 
#ifndef NDEBUG
std::string ver = "_d";
#else
std::string ver = "";
#endif
 
#include <iostream>
 
namespace bf = std::filesystem;
 
int main(int argc, char** argv)
{
   std::string usage = "Usage: \n";
   usage += "[path to fhsim_extras/tests] [screenverbosity] [logfileverbosity] ";
   usage += "[change visualization settings(y/n)] [path to FhSim binaries (opt)]";
   usage += "\nVerbosity: '0' - Nolog, '1' - Error, '2' - Warning, '3' - Info, '4' - Debug";
 
   if (argc == 1) {
      std::cerr << usage << std::endl;
      return 1;
   }
 
   if (argc > 1) testTools::testDir = std::string(argv[1]);
   if (argc > 2) testTools::logLevelToScreen = atoi(argv[2]);
   if (argc > 3) testTools::logLevelToLog = atoi(argv[3]);
   bool force_ogreconfig = (argc > 4) && (std::string(argv[4]) == "y" || std::string(argv[4]) == "Y");
   if (argc > 5) bf::current_path(argv[5]);
 
   testTools::testDir = bf::absolute(testTools::testDir).string();
   testTools::inPath  = bf::path(testTools::testDir) / bf::path("in");
   testTools::expPath = testTools::inPath;
   auto outPath       = bf::path(testTools::testDir) / bf::path("out");
   testTools::resPath = outPath;
   testTools::logPath = outPath;
   testTools::rmsPath = outPath;
   testTools::runPath = bf::path(std::filesystem::current_path());
   bool outdirOk      = tool::AssureDir(outPath);
 
   bf::path ogreResources = testTools::runPath.parent_path() / bf::path("resources");
   if (force_ogreconfig) {
      // Save rename the user ogre settings so that the ogre gui will pop up asking for settings
      if (std::filesystem::exists(ogreResources / bf::path("ogre.cfg"))) {
         bf::rename(ogreResources / bf::path("ogre.cfg"), ogreResources / bf::path("save_ogre.cfg"));
      }
   }
 
   std::cout << "Input directory:  " << testTools::inPath << std::endl;
   std::cout << "Output directory: " << outPath << std::endl;
   std::cout << "Run directory:    " << testTools::runPath << std::endl;
 
   ::testing::InitGoogleTest(&argc, argv);
   int result = RUN_ALL_TESTS();
 
#ifdef WIN32
   Sleep(2000);
#endif
 
   if (force_ogreconfig) {
      // Restore the users ogre setting file
      if (std::filesystem::exists(ogreResources / bf::path("ogre.cfg"))) {
         std::filesystem::remove(ogreResources / bf::path("ogre.cfg"));
         bf::rename(ogreResources / bf::path("save_ogre.cfg"), ogreResources / bf::path("ogre.cfg"));
      }
   }
 
   return result;
}

Running SimObject tests

Build the project with FH_WITH_TESTS=ON, which is done if you provide the conan config -c tools.build:skip_tests=False. The test will be run as part of the conan build process. The test executable in FhSim core is available under tests/bin in your build folder, e.g. testFhVis or testFhSim. Navigate to this directory and run e.g. testFhVis .. 4 4 n. Run without arguments to get usage description. Tests in SimObject libraries are usually located in playpen/bin and follow the same usage pattern. Note that for SimObject libraries it is required to point the environment variable SFH_LICENSE_FILE to a valid license file before attempting to run the tests.