EAS Development for Mainline Linux

EAS development for mainline Linux takes place on LKML and Linaro eas-dev mailing lists where proposed patches are discussed with the community. Arm now provides a pre-integrated version of latest EAS patches for mainline, for upstream development use.

EAS mainline is a moving integration branch tracking tip sched/core, mainline release candidate or linux-pm testing and adding the latest versions of EAS related patch sets targeted at the mainline kernel. The patches may already be under discussion on relevant open source mailing lists or waiting here while their dependencies get resolved and merged upstream.

The integration branch comes as is with synthetic test case testing only. The intention is to have a single branch containing the full EAS picture for mainline Linux kernel developers interested in tracking progress. This EAS mainline integration branch is not suitable for product development. For product, please use the AOSP common kernel.

Source code

EAS mainline integration branches are made available regularly on:

        https://gitlab.arm.com/linux-arm/linux-power

The releases are tagged with the date e.g. -20201127 so old integrations are available for comparison

Latest: integration-20210104

Tree: eas/next/integration

Base: eas/next/integration-base

Features:

  1. CPPC FFH support using AMUs (v5.11-rc1)
  2. Misfit improvements for 3-gear

Features in Linux mainline:

  1. v4.13 - CPU Invariant Engine (CIE)
  2. v4.15 - Frequency Invariant Engine (FIE)
  3. v4.17 - Idle CPU Per-Entity Load-Tracking (PELT) update
  4. v4.17 - Util(ization) Est(imated)
  5. v4.20 - Misfit task, i.e. forcing migration of running tasks that do not fit on the CPU they are currently running on
  6. v4.20 - Runtime scheduler domain flag detection
  7. v5.0   - Per-cpu Energy Model (EM) and Energy Aware Scheduling (EAS)
  8. v5.3   - Tracepoints (PELT and over-utilzation)
  9. v5.3   - Util(ization) clamping (core and per-task interface)
  10. v5.4   - Util(ization) clamping (cgroup interface)
  11. v5.4   - Patch-set 'sched/fair: Reduce complexity of energy calculation'
  12. v5.5   - Thermal/Cpu Cooling as Energy Model (EM) user
  13. v5.7   - Activity Monitor Unit (AMU) support
  14. v5.8   - Streamline select_task_rq() & select_task_rq_fair() (partly (4/9 patches))
  15. v5.9   - (Devfreq) devices as Energy Model user
  16. v5.9   - SCHED_DEADLINE capacity awareness
  17. v5.10 - Instrument sched domain flags
  18. v5.10 - Cpufreq FIE cleanup

Platforms we test on:

  1. Arm Juno r0 (arm64)
  2. Arm Juno r2 (arm64)
  3. Hikey960 (arm64) (needs out-of-tree patches  (hikey960_mainline topic))
  4. DB845C (arm64) (needs out-of-tree patch (db845c_mainline topic))
  5. Arm TC2 (arm32)

Next work items:

  1. Pushing Features upstream
  2. 100% test pass rate on all platforms
  • Test Results (integration 20210104) 

    (1) juno-r0:

    testcase                                      old%   new%  delta%       pvalue fix_iter# 
    -----------------------------------------------------------------------------------------
    Invariance:test_load_correctness             12.4%   6.8%   -5.6%     4.75e-02           
    Invariance:test_util_correctness             12.8%   6.8%   -6.0%     3.43e-02           
    OneTaskCPUMigration:test_util_task_migration 28.4%  16.8%  -11.6%     2.66e-03           

    test/eas_behaviour (LISA-test) in 0:08:59 [BAD]
        EnergyModelWakeMigration:test_dmesg:          passed    250/250 (100.0%)
        EnergyModelWakeMigration:test_slack:          passed    250/250 (100.0%)
        EnergyModelWakeMigration:test_task_placement: passed    250/250 (100.0%)
        OneSmallTask:test_dmesg:                      passed    250/250 (100.0%)
        OneSmallTask:test_slack:                      passed    250/250 (100.0%)
        OneSmallTask:test_task_placement:             FAILED    1/250   (0.4%)
        RampDown:test_dmesg:                          passed    250/250 (100.0%)
        RampDown:test_slack:                          passed    250/250 (100.0%)
        RampDown:test_task_placement:                 passed    250/250 (100.0%)
        RampUp:test_dmesg:                            FAILED    1/250   (0.4%)
        RampUp:test_slack:                            passed    250/250 (100.0%)
        RampUp:test_task_placement:                   undecided 1/250   (0.4%)
        ThreeSmallTasks:test_dmesg:                   passed    250/250 (100.0%)
        ThreeSmallTasks:test_slack:                   passed    250/250 (100.0%)
        ThreeSmallTasks:test_task_placement:          passed    250/250 (100.0%)
        TwoBigTasks:test_dmesg:                       passed    250/250 (100.0%)
        TwoBigTasks:test_slack:                       passed    250/250 (100.0%)
        TwoBigTasks:test_task_placement:              passed    250/250 (100.0%)
        TwoBigThreeSmall:test_dmesg:                  passed    250/250 (100.0%)
        TwoBigThreeSmall:test_slack:                  passed    250/250 (100.0%)
        TwoBigThreeSmall:test_task_placement:         passed    250/250 (100.0%)

    test/load_tracking (LISA-test) in 0:27:08 [BAD]
        Invariance:iter_invariance_items[cpu=0@450000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@450000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@450000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@450000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@450000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@575000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@575000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@575000]:test_load_correctness:  FAILED 1/250   (0.4%)
        Invariance:iter_invariance_items[cpu=0@575000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@575000]:test_util_correctness:  FAILED 1/250   (0.4%)
        Invariance:iter_invariance_items[cpu=0@700000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@700000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@700000]:test_load_correctness:  FAILED 1/250   (0.4%)
        Invariance:iter_invariance_items[cpu=0@700000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@700000]:test_util_correctness:  FAILED 1/250   (0.4%)
        Invariance:iter_invariance_items[cpu=0@775000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@775000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@775000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@775000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@775000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@850000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@850000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@850000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@850000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@850000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@450000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@450000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@450000]:test_load_correctness:  FAILED 15/250  (6.0%)
        Invariance:iter_invariance_items[cpu=1@450000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@450000]:test_util_correctness:  FAILED 15/250  (6.0%)
        Invariance:iter_invariance_items[cpu=1@625000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@625000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@625000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@625000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@625000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@800000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@800000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@800000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@800000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@800000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@950000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@950000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@950000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@950000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@950000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1100000]:test_dmesg:            passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1100000]:test_load_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1100000]:test_load_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1100000]:test_util_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1100000]:test_util_correctness: passed 250/250 (100.0%)
        Invariance:test_cpu_invariance:                                        passed 250/250 (100.0%)
        Invariance:test_freq_invariance:                                       passed 250/250 (100.0%)
        Invariance:test_load_behaviour:                                        passed 250/250 (100.0%)
        Invariance:test_load_correctness:                                      FAILED 17/250  (6.8%)
        Invariance:test_util_behaviour:                                        passed 250/250 (100.0%)
        Invariance:test_util_correctness:                                      FAILED 17/250  (6.8%)
        NTasksCPUMigration:test_dmesg:                                         passed 250/250 (100.0%)
        NTasksCPUMigration:test_util_task_migration:                           passed 250/250 (100.0%)
        OneTaskCPUMigration:test_dmesg:                                        passed 250/250 (100.0%)
        OneTaskCPUMigration:test_util_task_migration:                          FAILED 42/250  (16.8%)
        TwoTasksCPUMigration:test_dmesg:                                       passed 250/250 (100.0%)
        TwoTasksCPUMigration:test_util_task_migration:                         passed 250/250 (100.0%)

    test/misfit (LISA-test) in 0:02:20 [BAD]
        StaggeredFinishes:test_dmesg:        FAILED 6/250   (2.4%)
        StaggeredFinishes:test_preempt_time: passed 250/250 (100.0%)
        StaggeredFinishes:test_throughput:   passed 250/250 (100.0%)
        
    test/util_tracking (LISA-test) in 0:02:43 [BAD]
        UtilConvergence:test_activations: passed 250/250 (100.0%)
        UtilConvergence:test_dmesg:       passed 250/250 (100.0%)
        UtilConvergence:test_means:       FAILED 1/250   (0.4%)
        
    test/other (LISA-test) in 0:04:42 [BAD]
        CapacitySanity:test_capacity_sanity:     passed 250/250 (100.0%)
        UserspaceSanity:test_dmesg:              FAILED 5/250   (2.0%)
        UserspaceSanity:test_performance_sanity: passed 250/250 (100.0%)
        
    test/hotplug (LISA-test) in 0:01:57 [BAD]
        HotplugTorture:test_cpus_alive:   passed 250/250 (100.0%)
        HotplugTorture:test_dmesg:        FAILED 35/250  (14.0%)
        HotplugTorture:test_target_alive: passed 250/250 (100.0%)

    (2) juno-r2:

    testcase                                                              old%   new%  delta%       pvalue fix_iter# 
    -----------------------------------------------------------------------------------------------------------------
    Invariance:iter_invariance_items[cpu=1@600000]:test_load_correctness  7.9%   2.8%   -5.1%     2.83e-02           
    Invariance:iter_invariance_items[cpu=1@600000]:test_util_correctness  7.9%   3.2%   -4.7%     5.64e-02           
    Invariance:test_load_correctness                                      7.9%   2.8%   -5.1%     2.83e-02           
    Invariance:test_util_correctness                                      7.9%   3.2%   -4.7%     5.64e-02           
    TwoBigThreeSmall:test_task_placement                                 32.9%  22.8%  -10.1%     2.77e-02           

    test/eas_behaviour (LISA-test) in 0:08:50 [BAD]
        EnergyModelWakeMigration:test_dmesg:          passed    250/250 (100.0%)
        EnergyModelWakeMigration:test_slack:          passed    250/250 (100.0%)
        EnergyModelWakeMigration:test_task_placement: FAILED    2/250   (0.8%)
        OneSmallTask:test_dmesg:                      passed    250/250 (100.0%)
        OneSmallTask:test_slack:                      passed    250/250 (100.0%)
        OneSmallTask:test_task_placement:             FAILED    1/250   (0.4%)
        RampDown:test_dmesg:                          passed    250/250 (100.0%)
        RampDown:test_slack:                          FAILED    1/250   (0.4%)
        RampDown:test_task_placement:                 passed    250/250 (100.0%)
        RampUp:test_dmesg:                            passed    250/250 (100.0%)
        RampUp:test_slack:                            passed    250/250 (100.0%)
        RampUp:test_task_placement:                   passed    250/250 (100.0%)
        ThreeSmallTasks:test_dmesg:                   passed    250/250 (100.0%)
        ThreeSmallTasks:test_slack:                   passed    250/250 (100.0%)
        ThreeSmallTasks:test_task_placement:          passed    250/250 (100.0%)
        TwoBigTasks:test_dmesg:                       FAILED    1/250   (0.4%)
        TwoBigTasks:test_slack:                       passed    250/250 (100.0%)
        TwoBigTasks:test_task_placement:              undecided 1/250   (0.4%)
        TwoBigThreeSmall:test_dmesg:                  passed    250/250 (100.0%)
        TwoBigThreeSmall:test_slack:                  passed    250/250 (100.0%)
        TwoBigThreeSmall:test_task_placement:         FAILED    57/250  (22.8%)
        

    test/load_tracking (LISA-test) in 0:19:22 [BAD]
        Invariance:iter_invariance_items[cpu=0@450000]:test_dmesg:             passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=0@450000]:test_load_behaviour:    passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=0@450000]:test_load_correctness:  passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=0@450000]:test_util_behaviour:    passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=0@450000]:test_util_correctness:  passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=0@800000]:test_dmesg:             passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=0@800000]:test_load_behaviour:    passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=0@800000]:test_load_correctness:  passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=0@800000]:test_util_behaviour:    passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=0@800000]:test_util_correctness:  passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=0@950000]:test_dmesg:             passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=0@950000]:test_load_behaviour:    passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=0@950000]:test_load_correctness:  passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=0@950000]:test_util_behaviour:    passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=0@950000]:test_util_correctness:  passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=1@600000]:test_dmesg:             passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=1@600000]:test_load_behaviour:    passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=1@600000]:test_load_correctness:  FAILED 7/247   (2.8%)
        Invariance:iter_invariance_items[cpu=1@600000]:test_util_behaviour:    passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=1@600000]:test_util_correctness:  FAILED 8/247   (3.2%)
        Invariance:iter_invariance_items[cpu=1@1000000]:test_dmesg:            passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1000000]:test_load_behaviour:   passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1000000]:test_load_correctness: passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1000000]:test_util_behaviour:   passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1000000]:test_util_correctness: passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1200000]:test_dmesg:            passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1200000]:test_load_behaviour:   passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1200000]:test_load_correctness: passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1200000]:test_util_behaviour:   passed 247/247 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1200000]:test_util_correctness: passed 247/247 (100.0%)
        Invariance:test_cpu_invariance:                                        passed 247/247 (100.0%)
        Invariance:test_freq_invariance:                                       passed 247/247 (100.0%)
        Invariance:test_load_behaviour:                                        passed 247/247 (100.0%)
        Invariance:test_load_correctness:                                      FAILED 7/247   (2.8%)
        Invariance:test_util_behaviour:                                        passed 247/247 (100.0%)
        Invariance:test_util_correctness:                                      FAILED 8/247   (3.2%)
        NTasksCPUMigration:test_dmesg:                                         passed 248/248 (100.0%)
        NTasksCPUMigration:test_util_task_migration:                           passed 248/248 (100.0%)
        OneTaskCPUMigration:test_dmesg:                                        passed 248/248 (100.0%)
        OneTaskCPUMigration:test_util_task_migration:                          FAILED 12/248  (4.8%)
        TwoTasksCPUMigration:test_dmesg:                                       passed 248/248 (100.0%)
        TwoTasksCPUMigration:test_util_task_migration:                         passed 248/248 (100.0%)

    test/misfit (LISA-test) in 0:02:19 [BAD]
        StaggeredFinishes:test_dmesg:        FAILED 5/247   (2.0%)
        StaggeredFinishes:test_preempt_time: passed 247/247 (100.0%)
        StaggeredFinishes:test_throughput:   passed 247/247 (100.0%)
        
    test/util_tracking (LISA-test) in 0:02:41 [GOOD]
        UtilConvergence:test_activations: passed 249/249 (100.0%)
        UtilConvergence:test_dmesg:       passed 249/249 (100.0%)
        UtilConvergence:test_means:       passed 249/249 (100.0%)

    test/other (LISA-test) in 0:03:41 [BAD]
        CapacitySanity:test_capacity_sanity:     passed 247/247 (100.0%)
        UserspaceSanity:test_dmesg:              FAILED 2/247   (0.8%)
        UserspaceSanity:test_performance_sanity: passed 247/247 (100.0%)
        
    test/hotplug (LISA-test) in 0:01:57 [BAD]
        HotplugTorture:test_cpus_alive:   passed 250/250 (100.0%)
        HotplugTorture:test_dmesg:        FAILED 249/250 (99.6%)
        HotplugTorture:test_target_alive: passed 250/250 (100.0%)

    (3) hikey960:

    testcase                                     old%   new%  delta%       pvalue fix_iter# 
    ----------------------------------------------------------------------------------------
    OneSmallTask:test_task_placement             2.0%   0.0%   -2.0%     6.13e-02           
    OneTaskCPUMigration:test_util_task_migration 0.0%   3.2%    3.2%     7.38e-03       112 

    test/eas_behaviour (LISA-test) in 0:08:29 [GOOD]
        EnergyModelWakeMigration:test_dmesg:          passed 250/250 (100.0%)
        EnergyModelWakeMigration:test_slack:          passed 250/250 (100.0%)
        EnergyModelWakeMigration:test_task_placement: passed 250/250 (100.0%)
        OneSmallTask:test_dmesg:                      passed 250/250 (100.0%)
        OneSmallTask:test_slack:                      passed 250/250 (100.0%)
        OneSmallTask:test_task_placement:             passed 250/250 (100.0%)
        RampDown:test_dmesg:                          passed 250/250 (100.0%)
        RampDown:test_slack:                          passed 250/250 (100.0%)
        RampDown:test_task_placement:                 passed 250/250 (100.0%)
        RampUp:test_dmesg:                            passed 250/250 (100.0%)
        RampUp:test_slack:                            passed 250/250 (100.0%)
        RampUp:test_task_placement:                   passed 250/250 (100.0%)
        ThreeSmallTasks:test_dmesg:                   passed 250/250 (100.0%)
        ThreeSmallTasks:test_slack:                   passed 250/250 (100.0%)
        ThreeSmallTasks:test_task_placement:          passed 250/250 (100.0%)
        TwoBigTasks:test_dmesg:                       passed 250/250 (100.0%)
        TwoBigTasks:test_slack:                       passed 250/250 (100.0%)
        TwoBigTasks:test_task_placement:              passed 250/250 (100.0%)
        TwoBigThreeSmall:test_dmesg:                  passed 250/250 (100.0%)
        TwoBigThreeSmall:test_slack:                  passed 250/250 (100.0%)
        TwoBigThreeSmall:test_task_placement:         passed 250/250 (100.0%)
        
    test/load_tracking (LISA-test) in 0:29:50 [BAD]
        Invariance:iter_invariance_items[cpu=0@533000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@533000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@533000]:test_load_correctness:  FAILED 5/250   (2.0%)
        Invariance:iter_invariance_items[cpu=0@533000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@533000]:test_util_correctness:  FAILED 5/250   (2.0%)
        Invariance:iter_invariance_items[cpu=0@999000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@999000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@999000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@999000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@999000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1402000]:test_dmesg:            passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1402000]:test_load_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1402000]:test_load_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1402000]:test_util_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1402000]:test_util_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1709000]:test_dmesg:            passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1709000]:test_load_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1709000]:test_load_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1709000]:test_util_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1709000]:test_util_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1844000]:test_dmesg:            passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1844000]:test_load_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1844000]:test_load_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1844000]:test_util_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1844000]:test_util_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@903000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@903000]:test_load_behaviour:    FAILED 2/250   (0.8%)
        Invariance:iter_invariance_items[cpu=4@903000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@903000]:test_util_behaviour:    FAILED 2/250   (0.8%)
        Invariance:iter_invariance_items[cpu=4@903000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1421000]:test_dmesg:            passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1421000]:test_load_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1421000]:test_load_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1421000]:test_util_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1421000]:test_util_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1805000]:test_dmesg:            passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1805000]:test_load_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1805000]:test_load_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1805000]:test_util_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1805000]:test_util_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2112000]:test_dmesg:            passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2112000]:test_load_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2112000]:test_load_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2112000]:test_util_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2112000]:test_util_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2362000]:test_dmesg:            passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2362000]:test_load_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2362000]:test_load_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2362000]:test_util_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2362000]:test_util_correctness: passed 250/250 (100.0%)
        Invariance:test_cpu_invariance:                                        passed 250/250 (100.0%)
        Invariance:test_freq_invariance:                                       passed 250/250 (100.0%)
        Invariance:test_load_behaviour:                                        FAILED 2/250   (0.8%)
        Invariance:test_load_correctness:                                      FAILED 5/250   (2.0%)
        Invariance:test_util_behaviour:                                        FAILED 2/250   (0.8%)
        Invariance:test_util_correctness:                                      FAILED 5/250   (2.0%)
        NTasksCPUMigration:test_dmesg:                                         passed 250/250 (100.0%)
        NTasksCPUMigration:test_util_task_migration:                           passed 250/250 (100.0%)
        OneTaskCPUMigration:test_dmesg:                                        passed 250/250 (100.0%)
        OneTaskCPUMigration:test_util_task_migration:                          FAILED 8/250   (3.2%)
        TwoTasksCPUMigration:test_dmesg:                                       passed 250/250 (100.0%)
        TwoTasksCPUMigration:test_util_task_migration:                         FAILED 3/250   (1.2%)
        
    test/misfit (LISA-test) in 0:02:15 [GOOD]
        StaggeredFinishes:test_dmesg:        passed    250/250 (100.0%)
        StaggeredFinishes:test_preempt_time: passed    250/250 (100.0%)
        StaggeredFinishes:test_throughput:   undecided 84/250  (33.6%)
        
    test/util_tracking (LISA-test) in 0:02:35 [GOOD]
        UtilConvergence:test_activations: passed 250/250 (100.0%)
        UtilConvergence:test_dmesg:       passed 250/250 (100.0%)
        UtilConvergence:test_means:       passed 250/250 (100.0%)
        
    test/other (LISA-test) in 0:04:43 [GOOD]
        CapacitySanity:test_capacity_sanity:     passed 250/250 (100.0%)
        UserspaceSanity:test_dmesg:              passed 250/250 (100.0%)
        UserspaceSanity:test_performance_sanity: passed 250/250 (100.0%)

    (4) TC2:

    test/eas_behaviour (LISA-test) in 0:08:24 [BAD]
        EnergyModelWakeMigration:test_dmesg:          passed 250/250 (100.0%)
        EnergyModelWakeMigration:test_slack:          passed 250/250 (100.0%)
        EnergyModelWakeMigration:test_task_placement: passed 250/250 (100.0%)
        OneSmallTask:test_dmesg:                      passed 250/250 (100.0%)
        OneSmallTask:test_slack:                      passed 250/250 (100.0%)
        OneSmallTask:test_task_placement:             passed 250/250 (100.0%)
        RampDown:test_dmesg:                          passed 250/250 (100.0%)
        RampDown:test_slack:                          passed 250/250 (100.0%)
        RampDown:test_task_placement:                 passed 250/250 (100.0%)
        RampUp:test_dmesg:                            passed 250/250 (100.0%)
        RampUp:test_slack:                            passed 250/250 (100.0%)
        RampUp:test_task_placement:                   passed 250/250 (100.0%)
        ThreeSmallTasks:test_dmesg:                   passed 250/250 (100.0%)
        ThreeSmallTasks:test_slack:                   passed 250/250 (100.0%)
        ThreeSmallTasks:test_task_placement:          passed 250/250 (100.0%)
        TwoBigTasks:test_dmesg:                       passed 250/250 (100.0%)
        TwoBigTasks:test_slack:                       passed 250/250 (100.0%)
        TwoBigTasks:test_task_placement:              passed 250/250 (100.0%)
        TwoBigThreeSmall:test_dmesg:                  passed 250/250 (100.0%)
        TwoBigThreeSmall:test_slack:                  passed 250/250 (100.0%)
        TwoBigThreeSmall:test_task_placement:         FAILED 4/250   (1.6%)

    test/load_tracking (LISA-test) in 0:33:37 [GOOD]
        Invariance:iter_invariance_items[cpu=0@350000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@350000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@350000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@350000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@350000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@400000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@400000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@400000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@400000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@400000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@500000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@500000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@500000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@500000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@500000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@600000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@600000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@600000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@600000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@600000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@700000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@700000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@700000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@700000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@700000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@800000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@800000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@800000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@800000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@800000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@900000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@900000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@900000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@900000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@900000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1000000]:test_dmesg:            passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1000000]:test_load_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1000000]:test_load_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1000000]:test_util_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1000000]:test_util_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@500000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@500000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@500000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@500000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@500000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@600000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@600000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@600000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@600000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@600000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@700000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@700000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@700000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@700000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@700000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@800000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@800000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@800000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@800000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@800000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@900000]:test_dmesg:             passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@900000]:test_load_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@900000]:test_load_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@900000]:test_util_behaviour:    passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@900000]:test_util_correctness:  passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1000000]:test_dmesg:            passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1000000]:test_load_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1000000]:test_load_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1000000]:test_util_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1000000]:test_util_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1100000]:test_dmesg:            passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1100000]:test_load_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1100000]:test_load_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1100000]:test_util_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1100000]:test_util_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1200000]:test_dmesg:            passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1200000]:test_load_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1200000]:test_load_correctness: passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1200000]:test_util_behaviour:   passed 250/250 (100.0%)
        Invariance:iter_invariance_items[cpu=1@1200000]:test_util_correctness: passed 250/250 (100.0%)
        Invariance:test_cpu_invariance:                                        passed 250/250 (100.0%)
        Invariance:test_freq_invariance:                                       passed 250/250 (100.0%)
        Invariance:test_load_behaviour:                                        passed 250/250 (100.0%)
        Invariance:test_load_correctness:                                      passed 250/250 (100.0%)
        Invariance:test_util_behaviour:                                        passed 250/250 (100.0%)
        Invariance:test_util_correctness:                                      passed 250/250 (100.0%)
        NTasksCPUMigration:test_dmesg:                                         passed 250/250 (100.0%)
        NTasksCPUMigration:test_util_task_migration:                           passed 250/250 (100.0%)
        OneTaskCPUMigration:test_dmesg:                                        passed 250/250 (100.0%)
        OneTaskCPUMigration:test_util_task_migration:                          passed 250/250 (100.0%)
        TwoTasksCPUMigration:test_dmesg:                                       passed 250/250 (100.0%)
        TwoTasksCPUMigration:test_util_task_migration:                         passed 250/250 (100.0%)
        
    test/misfit (LISA-test) in 0:02:08 [BAD]
        StaggeredFinishes:test_dmesg:        passed 250/250 (100.0%)
        StaggeredFinishes:test_preempt_time: passed 250/250 (100.0%)
        StaggeredFinishes:test_throughput:   FAILED 1/250   (0.4%)
        
    test/util_tracking (LISA-test) in 0:02:27 [GOOD]
        UtilConvergence:test_activations: passed 250/250 (100.0%)
        UtilConvergence:test_dmesg:       passed 250/250 (100.0%)
        UtilConvergence:test_means:       passed 250/250 (100.0%)
        
    test/other (LISA-test) in 0:05:38 [GOOD]
        CapacitySanity:test_capacity_sanity:     passed 250/250 (100.0%)
        UserspaceSanity:test_dmesg:              passed 250/250 (100.0%)
        UserspaceSanity:test_performance_sanity: passed 250/250 (100.0%)

    (5) db845c:

    testcase                                                               old%   new%  delta%       pvalue fix_iter# 
    ------------------------------------------------------------------------------------------------------------------
    Invariance:iter_invariance_items[cpu=0@1324800]:test_load_correctness  0.0%   2.4%    2.4%     6.12e-02       141 
    Invariance:iter_invariance_items[cpu=0@1324800]:test_util_correctness  0.0%   2.4%    2.4%     6.12e-02       141 
    Invariance:iter_invariance_items[cpu=0@1689600]:test_load_behaviour    3.0%   0.0%   -3.0%     1.34e-02           
    Invariance:iter_invariance_items[cpu=0@1689600]:test_load_correctness  2.5%   0.0%   -2.5%     2.76e-02           
    Invariance:iter_invariance_items[cpu=0@1689600]:test_util_behaviour    3.0%   0.0%   -3.0%     1.34e-02           
    Invariance:iter_invariance_items[cpu=0@1689600]:test_util_correctness  2.5%   0.0%   -2.5%     2.76e-02           
    Invariance:iter_invariance_items[cpu=0@1766400]:test_load_behaviour    3.0%   0.0%   -3.0%     1.34e-02           
    Invariance:iter_invariance_items[cpu=0@1766400]:test_util_behaviour    3.0%   0.0%   -3.0%     1.34e-02           
    Invariance:iter_invariance_items[cpu=0@300000]:test_load_correctness   0.0%   5.7%    5.7%     4.25e-04        56 
    Invariance:iter_invariance_items[cpu=0@825600]:test_load_correctness  10.3%   0.0%  -10.3%     1.82e-07           
    Invariance:iter_invariance_items[cpu=0@825600]:test_util_correctness  10.3%   0.0%  -10.3%     1.82e-07           
    Invariance:test_cpu_invariance                                         3.0%   0.0%   -3.0%     1.34e-02           
    Invariance:test_load_behaviour                                         5.9%   0.0%   -5.9%     1.63e-04           
    Invariance:test_util_behaviour                                         5.9%   0.0%   -5.9%     1.63e-04           
    ThreeSmallTasks:test_task_placement                                    0.0%   5.7%    5.7%     4.25e-04        56 
    UtilConvergence:test_means                                            77.3%   0.0%  -77.3%     1.17e-72           

    test/eas_behaviour (LISA-test) in 0:10:25 [BAD]
        EnergyModelWakeMigration:test_dmesg:          passed    211/211 (100.0%)
        EnergyModelWakeMigration:test_slack:          passed    211/211 (100.0%)
        EnergyModelWakeMigration:test_task_placement: undecided 188/211 (89.1%)
        OneSmallTask:test_dmesg:                      passed    211/211 (100.0%)
        OneSmallTask:test_slack:                      passed    211/211 (100.0%)
        OneSmallTask:test_task_placement:             passed    211/211 (100.0%)
        RampDown:test_dmesg:                          passed    211/211 (100.0%)
        RampDown:test_slack:                          passed    211/211 (100.0%)
        RampDown:test_task_placement:                 undecided 177/211 (83.9%)
        RampUp:test_dmesg:                            passed    211/211 (100.0%)
        RampUp:test_slack:                            passed    211/211 (100.0%)
        RampUp:test_task_placement:                   undecided 152/211 (72.0%)
        ThreeSmallTasks:test_dmesg:                   passed    211/211 (100.0%)
        ThreeSmallTasks:test_slack:                   passed    211/211 (100.0%)
        ThreeSmallTasks:test_task_placement:          FAILED    12/211  (5.7%)
        TwoBigTasks:test_dmesg:                       passed    211/211 (100.0%)
        TwoBigTasks:test_slack:                       passed    211/211 (100.0%)
        TwoBigTasks:test_task_placement:              undecided 190/211 (90.0%)
        TwoBigThreeSmall:test_dmesg:                  passed    211/211 (100.0%)
        TwoBigThreeSmall:test_slack:                  passed    211/211 (100.0%)
        TwoBigThreeSmall:test_task_placement:         undecided 1/211   (0.5%)
        
    test/load_tracking (LISA-test) in 0:42:25 [BAD]
        Invariance:iter_invariance_items[cpu=0@300000]:test_dmesg:             passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@300000]:test_load_behaviour:    undecided 42/211  (19.9%)
        Invariance:iter_invariance_items[cpu=0@300000]:test_load_correctness:  FAILED    12/211  (5.7%)
        Invariance:iter_invariance_items[cpu=0@300000]:test_util_behaviour:    undecided 42/211  (19.9%)
        Invariance:iter_invariance_items[cpu=0@300000]:test_util_correctness:  passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@576000]:test_dmesg:             passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@576000]:test_load_behaviour:    undecided 2/211   (0.9%)
        Invariance:iter_invariance_items[cpu=0@576000]:test_load_correctness:  FAILED    70/211  (33.2%)
        Invariance:iter_invariance_items[cpu=0@576000]:test_util_behaviour:    undecided 2/211   (0.9%)
        Invariance:iter_invariance_items[cpu=0@576000]:test_util_correctness:  FAILED    73/211  (34.6%)
        Invariance:iter_invariance_items[cpu=0@825600]:test_dmesg:             passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@825600]:test_load_behaviour:    passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@825600]:test_load_correctness:  passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@825600]:test_util_behaviour:    passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@825600]:test_util_correctness:  passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1056000]:test_dmesg:            passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1056000]:test_load_behaviour:   passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1056000]:test_load_correctness: FAILED    7/211   (3.3%)
        Invariance:iter_invariance_items[cpu=0@1056000]:test_util_behaviour:   passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1056000]:test_util_correctness: FAILED    7/211   (3.3%)
        Invariance:iter_invariance_items[cpu=0@1324800]:test_dmesg:            passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1324800]:test_load_behaviour:   passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1324800]:test_load_correctness: FAILED    5/211   (2.4%)
        Invariance:iter_invariance_items[cpu=0@1324800]:test_util_behaviour:   passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1324800]:test_util_correctness: FAILED    5/211   (2.4%)
        Invariance:iter_invariance_items[cpu=0@1516800]:test_dmesg:            passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1516800]:test_load_behaviour:   passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1516800]:test_load_correctness: FAILED    1/211   (0.5%)
        Invariance:iter_invariance_items[cpu=0@1516800]:test_util_behaviour:   passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1516800]:test_util_correctness: FAILED    1/211   (0.5%)
        Invariance:iter_invariance_items[cpu=0@1689600]:test_dmesg:            passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1689600]:test_load_behaviour:   passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1689600]:test_load_correctness: passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1689600]:test_util_behaviour:   passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1689600]:test_util_correctness: passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1766400]:test_dmesg:            passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1766400]:test_load_behaviour:   passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1766400]:test_load_correctness: passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1766400]:test_util_behaviour:   passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=0@1766400]:test_util_correctness: passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@825600]:test_dmesg:             passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@825600]:test_load_behaviour:    undecided 25/211  (11.8%)
        Invariance:iter_invariance_items[cpu=4@825600]:test_load_correctness:  passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@825600]:test_util_behaviour:    undecided 25/211  (11.8%)
        Invariance:iter_invariance_items[cpu=4@825600]:test_util_correctness:  passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1056000]:test_dmesg:            passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1056000]:test_load_behaviour:   undecided 24/211  (11.4%)
        Invariance:iter_invariance_items[cpu=4@1056000]:test_load_correctness: passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1056000]:test_util_behaviour:   undecided 24/211  (11.4%)
        Invariance:iter_invariance_items[cpu=4@1056000]:test_util_correctness: passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1459200]:test_dmesg:            passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1459200]:test_load_behaviour:   undecided 20/211  (9.5%)
        Invariance:iter_invariance_items[cpu=4@1459200]:test_load_correctness: passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1459200]:test_util_behaviour:   undecided 20/211  (9.5%)
        Invariance:iter_invariance_items[cpu=4@1459200]:test_util_correctness: passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1766400]:test_dmesg:            passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1766400]:test_load_behaviour:   undecided 17/211  (8.1%)
        Invariance:iter_invariance_items[cpu=4@1766400]:test_load_correctness: passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1766400]:test_util_behaviour:   undecided 17/211  (8.1%)
        Invariance:iter_invariance_items[cpu=4@1766400]:test_util_correctness: passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1996800]:test_dmesg:            passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1996800]:test_load_behaviour:   undecided 18/211  (8.5%)
        Invariance:iter_invariance_items[cpu=4@1996800]:test_load_correctness: passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@1996800]:test_util_behaviour:   undecided 18/211  (8.5%)
        Invariance:iter_invariance_items[cpu=4@1996800]:test_util_correctness: passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2323200]:test_dmesg:            passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2323200]:test_load_behaviour:   undecided 21/211  (10.0%)
        Invariance:iter_invariance_items[cpu=4@2323200]:test_load_correctness: passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2323200]:test_util_behaviour:   undecided 21/211  (10.0%)
        Invariance:iter_invariance_items[cpu=4@2323200]:test_util_correctness: passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2553600]:test_dmesg:            passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2553600]:test_load_behaviour:   undecided 11/211  (5.2%)
        Invariance:iter_invariance_items[cpu=4@2553600]:test_load_correctness: passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2553600]:test_util_behaviour:   undecided 11/211  (5.2%)
        Invariance:iter_invariance_items[cpu=4@2553600]:test_util_correctness: passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2649600]:test_dmesg:            passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2649600]:test_load_behaviour:   undecided 18/211  (8.5%)
        Invariance:iter_invariance_items[cpu=4@2649600]:test_load_correctness: passed    211/211 (100.0%)
        Invariance:iter_invariance_items[cpu=4@2649600]:test_util_behaviour:   undecided 18/211  (8.5%)
        Invariance:iter_invariance_items[cpu=4@2649600]:test_util_correctness: passed    211/211 (100.0%)
        Invariance:test_cpu_invariance:                                        passed    211/211 (100.0%)
        Invariance:test_freq_invariance:                                       passed    211/211 (100.0%)
        Invariance:test_load_behaviour:                                        passed    211/211 (100.0%)
        Invariance:test_load_correctness:                                      FAILED    87/211  (41.2%)
        Invariance:test_util_behaviour:                                        passed    211/211 (100.0%)
        Invariance:test_util_correctness:                                      FAILED    81/211  (38.4%)
        NTasksCPUMigration:test_dmesg:                                         passed    211/211 (100.0%)
        NTasksCPUMigration:test_util_task_migration:                           passed    211/211 (100.0%)
        OneTaskCPUMigration:test_dmesg:                                        passed    211/211 (100.0%)
        OneTaskCPUMigration:test_util_task_migration:                          passed    211/211 (100.0%)
        TwoTasksCPUMigration:test_dmesg:                                       passed    211/211 (100.0%)
        TwoTasksCPUMigration:test_util_task_migration:                         passed    211/211 (100.0%)
        
    test/misfit (LISA-test) in 0:02:51 [GOOD]
        StaggeredFinishes:test_dmesg:        passed    211/211 (100.0%)
        StaggeredFinishes:test_preempt_time: passed    211/211 (100.0%)
        StaggeredFinishes:test_throughput:   undecided 202/211 (95.7%)
        
    test/util_tracking (LISA-test) in 0:03:13 [GOOD]
        UtilConvergence:test_activations: passed 211/211 (100.0%)
        UtilConvergence:test_dmesg:       passed 211/211 (100.0%)
        UtilConvergence:test_means:       passed 211/211 (100.0%)
        
    test/other (LISA-test) in 0:06:23 [BAD]
        CapacitySanity:test_capacity_sanity:     passed 211/211 (100.0%)
        UserspaceSanity:test_dmesg:              passed 211/211 (100.0%)
        UserspaceSanity:test_performance_sanity: FAILED 8/211   (3.8%)
        
    test/hotplug (LISA-test) in 0:02:29 [BAD]
        HotplugTorture:test_cpus_alive:   passed 211/211 (100.0%)
        HotplugTorture:test_dmesg:        FAILED 211/211 (100.0%)
        HotplugTorture:test_target_alive: passed 211/211 (100.0%)