EAS Development for Mainline Linux

EAS development for mainline Linux takes place on LKML and Linaro eas-dev mailing lists where proposed patches are discussed with the community. Arm now provides a pre-integrated version of latest EAS patches for mainline, for upstream development use.

EAS mainline is a moving integration branch tracking tip sched/core, mainline release candidate or linux-pm testing and adding the latest versions of EAS related patch sets targeted at the mainline kernel. The patches may already be under discussion on relevant open source mailing lists or waiting here while their dependencies get resolved and merged upstream.

The integration branch comes as is with synthetic test case testing only. The intention is to have a single branch containing the full EAS picture for mainline Linux kernel developers interested in tracking progress. This EAS mainline integration branch is not suitable for product development. For product, please use the AOSP common kernel.

Source code

EAS mainline integration branches are made available regularly on:

        git://linux-arm.org/linux-power.git

The releases are tagged with the date e.g. -20190310 so old integrations are available for comparison

Latest: integration-20191209

Tree: eas/next/integration

Base: eas/next/integration-base

New changes:

Features:

  1. Schedutil as EM user (v3 on lkml)
  2. Util(ization) update during Idle migration (v1 on lkml)
  3. Activity Monitor Unit (AMU) support (v2 on lkml)

Features in Linux v5.5-rc1:

  1. Intelligent Power Allocation (IPA) as EM (Energy Model) user

Features in Linux mainline:

  1. v4.13 - CPU Invariant Engine (CIE)
  2. v4.15 - Frequency Invariant Engine (FIE)
  3. v4.17 - Idle CPU PELT (Per-Entity Load-Tracking) update
  4. v4.17 - Util(ization) Est(imated)
  5. v4.20 - Misfit task, i.e. forcing migration of running tasks that do not fit on the CPU they are currently running on
  6. v4.20 - Runtime scheduler domain flag detection
  7. v5.0   - Per-cpu Energy Model and Energy Aware Scheduling
  8. v5.3   - Tracepoints (PELT and over-utilzation)
  9. v5.3   - Util(ization) clamping (core and per-task interface)
  10. v5.4   - Util(ization) clamping (cgroup interface)
  11. v5.4   - Patch-set 'sched/fair: Reduce complexity of energy calculation'

Platforms we test on:

  1. Arm Juno r0 (arm64)
  2. Arm Juno r2 (arm64)
  3. Hikey960 (arm64) (needs out-of-tree patches  (hikey960_mainline topic))
  4. Arm TC2 (arm32) (needs out-of-tree patch (TC2_mainline topic))
  5. Pixel3 (arm64) (needs out-of-tree patch (pixel3_mainline topic))

Next work items:

  1. Pushing Features upstream
  2. 100% test pass rate on all platforms
  • Test Results (integration 20191209) 

    (1) juno-r0:

        testcase                                                                old%   new%  delta%       pvalue fix_iter# 
        -------------------------------------------------------------------------------------------------------------------
        OneSmallTask:test_task_placement                                        3.4%  19.8%   16.4%     1.04e-07        14 
        OneTaskCPUMigration:test_util_task_migration                            0.8%  59.8%   59.0%     4.13e-56         3 
        StaggeredFinishes:test_throughput                                      14.5%   0.4%  -14.1%     7.20e-11           
        TwoBigThreeSmall:test_task_placement                                    0.0%   6.0%    6.0%     4.95e-05        50 
        
        Description: Linux_version_5.5.0-rc1-gd394df8202_(pdswauto@oss-arch-dev-jenkins-agent-02)_(gcc_version_8.3.0_(GNU_Toolchain_for_the_A-profile_Architecture_8.3-2019.03_(arm-rel-8.36)))__with_board_juno
        
        test/eas_behaviour (LISA-test) in 0:06:24 [BAD]
            EnergyModelWakeMigration[board=juno-r0/00:02:f7:00:58:21]:test_slack:          passed    249/249 (100.0%)
            EnergyModelWakeMigration[board=juno-r0/00:02:f7:00:58:21]:test_task_placement: passed    249/249 (100.0%)
            OneSmallTask[board=juno-r0/00:02:f7:00:58:21]:test_slack:                      passed    249/249 (100.0%)
            OneSmallTask[board=juno-r0/00:02:f7:00:58:21]:test_task_placement:             undecided 47/249  (18.9%)
            OneSmallTask[board=juno-r0/00:02:f7:00:58:21]:test_task_placement:             FAILED    40/249  (16.1%)
            RampDown[board=juno-r0/00:02:f7:00:58:21]:test_slack:                          passed    249/249 (100.0%)
            RampDown[board=juno-r0/00:02:f7:00:58:21]:test_task_placement:                 undecided 8/249   (3.2%)
            RampUp[board=juno-r0/00:02:f7:00:58:21]:test_slack:                            passed    249/249 (100.0%)
            RampUp[board=juno-r0/00:02:f7:00:58:21]:test_task_placement:                   undecided 12/249  (4.8%)
            ThreeSmallTasks[board=juno-r0/00:02:f7:00:58:21]:test_slack:                   passed    249/249 (100.0%)
            ThreeSmallTasks[board=juno-r0/00:02:f7:00:58:21]:test_task_placement:          undecided 112/249 (45.0%)
            ThreeSmallTasks[board=juno-r0/00:02:f7:00:58:21]:test_task_placement:          FAILED    1/249   (0.4%)
            TwoBigTasks[board=juno-r0/00:02:f7:00:58:21]:test_slack:                       passed    249/249 (100.0%)
            TwoBigTasks[board=juno-r0/00:02:f7:00:58:21]:test_task_placement:              passed    249/249 (100.0%)
            TwoBigThreeSmall[board=juno-r0/00:02:f7:00:58:21]:test_slack:                  passed    249/249 (100.0%)
            TwoBigThreeSmall[board=juno-r0/00:02:f7:00:58:21]:test_task_placement:         undecided 1/249   (0.4%)
            TwoBigThreeSmall[board=juno-r0/00:02:f7:00:58:21]:test_task_placement:         FAILED    15/249  (6.0%)
            
            Error: 0/14, Failed: 3/14, Undecided: 5/14, Skipped: 0/14, Passed: 9/14
        
        test/load_tracking (LISA-test) in 0:13:15 [BAD]
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@450000]:test_load_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@450000]:test_load_correctness:  passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@450000]:test_util_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@450000]:test_util_correctness:  passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@575000]:test_load_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@575000]:test_load_correctness:  passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@575000]:test_util_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@575000]:test_util_correctness:  passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@700000]:test_load_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@700000]:test_load_correctness:  passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@700000]:test_util_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@700000]:test_util_correctness:  passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@775000]:test_load_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@775000]:test_load_correctness:  passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@775000]:test_util_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@775000]:test_util_correctness:  passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@850000]:test_load_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@850000]:test_load_correctness:  passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@850000]:test_util_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=0@850000]:test_util_correctness:  passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@450000]:test_load_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@450000]:test_load_correctness:  passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@450000]:test_util_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@450000]:test_util_correctness:  passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@625000]:test_load_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@625000]:test_load_correctness:  FAILED 11/249  (4.4%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@625000]:test_util_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@625000]:test_util_correctness:  FAILED 11/249  (4.4%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@800000]:test_load_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@800000]:test_load_correctness:  passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@800000]:test_util_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@800000]:test_util_correctness:  passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@950000]:test_load_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@950000]:test_load_correctness:  passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@950000]:test_util_behaviour:    passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@950000]:test_util_correctness:  passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@1100000]:test_load_behaviour:   passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@1100000]:test_load_correctness: passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@1100000]:test_util_behaviour:   passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:iter_invariance_items[cpu=1@1100000]:test_util_correctness: passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:test_cpu_invariance:                                        passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:test_freq_invariance:                                       passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:test_load_behaviour:                                        passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:test_load_correctness:                                      FAILED 11/249  (4.4%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:test_util_behaviour:                                        passed 249/249 (100.0%)
            Invariance[board=juno-r0/00:02:f7:00:58:21]:test_util_correctness:                                      FAILED 11/249  (4.4%)
            NTasksCPUMigration[board=juno-r0/00:02:f7:00:58:21]:test_util_task_migration:                           passed 249/249 (100.0%)
            OneTaskCPUMigration[board=juno-r0/00:02:f7:00:58:21]:test_util_task_migration:                          FAILED 149/249 (59.8%)
            TwoTasksCPUMigration[board=juno-r0/00:02:f7:00:58:21]:test_util_task_migration:                         passed 249/249 (100.0%)
            
            Error: 0/49, Failed: 5/49, Undecided: 0/49, Skipped: 0/49, Passed: 44/49
        
        test/misfit (LISA-test) in 0:01:57 [BAD]
            StaggeredFinishes[board=juno-r0/00:02:f7:00:58:21]:test_preempt_time: passed 249/249 (100.0%)
            StaggeredFinishes[board=juno-r0/00:02:f7:00:58:21]:test_throughput:   FAILED 1/249   (0.4%)
            
            Error: 0/2, Failed: 1/2, Undecided: 0/2, Skipped: 0/2, Passed: 1/2
        
        test/other (LISA-test) in 0:04:17 [GOOD]
            CapacitySanity[board=juno-r0/00:02:f7:00:58:21]:test_capacity_sanity:     passed 249/249 (100.0%)
            UserspaceSanity[board=juno-r0/00:02:f7:00:58:21]:test_performance_sanity: passed 249/249 (100.0%)
            
            Error: 0/2, Failed: 0/2, Undecided: 0/2, Skipped: 0/2, Passed: 2/2
        
        test/hotplug (LISA-test) in 0:01:47 [GOOD]
            HotplugTorture[board=juno-r0/00:02:f7:00:58:21]:test_cpus_alive:   passed 249/249 (100.0%)
            HotplugTorture[board=juno-r0/00:02:f7:00:58:21]:test_target_alive: passed 249/249 (100.0%)
            
            Error: 0/2, Failed: 0/2, Undecided: 0/2, Skipped: 0/2, Passed: 2/2
        
    (2) juno-r2:

        testcase                                                                old%   new%  delta%       pvalue fix_iter# 
        -------------------------------------------------------------------------------------------------------------------
        Invariance:iter_invariance_items[cpu=0@950000]:test_load_correctness    1.0%   7.6%    6.6%     1.87e-02        39 
        Invariance:iter_invariance_items[cpu=0@950000]:test_util_correctness    0.0%   7.6%    7.6%     2.66e-03        39 
        Invariance:test_util_correctness                                        3.1%  10.8%    7.7%     1.94e-02        26 
        OneTaskCPUMigration:test_util_task_migration                           52.0%  15.2%  -36.8%     9.68e-12           
        StaggeredFinishes:test_throughput                                      12.4%   0.0%  -12.4%     1.36e-07           
        
        Description: Linux_version_5.5.0-rc1-gd394df8202_(pdswauto@oss-arch-dev-jenkins-agent-04)_(gcc_version_8.3.0_(GNU_Toolchain_for_the_A-profile_Architecture_8.3-2019.03_(arm-rel-8.36)))__with_board_juno
        
        test/eas_behaviour (LISA-test) in 0:06:15 [GOOD]
            EnergyModelWakeMigration[board=juno-r2/00:02:f7:00:67:93]:test_slack:          passed    250/250 (100.0%)
            EnergyModelWakeMigration[board=juno-r2/00:02:f7:00:67:93]:test_task_placement: passed    250/250 (100.0%)
            OneSmallTask[board=juno-r2/00:02:f7:00:67:93]:test_slack:                      passed    250/250 (100.0%)
            OneSmallTask[board=juno-r2/00:02:f7:00:67:93]:test_task_placement:             undecided 3/250   (1.2%)
            RampDown[board=juno-r2/00:02:f7:00:67:93]:test_slack:                          passed    250/250 (100.0%)
            RampDown[board=juno-r2/00:02:f7:00:67:93]:test_task_placement:                 passed    250/250 (100.0%)
            RampUp[board=juno-r2/00:02:f7:00:67:93]:test_slack:                            passed    250/250 (100.0%)
            RampUp[board=juno-r2/00:02:f7:00:67:93]:test_task_placement:                   passed    250/250 (100.0%)
            ThreeSmallTasks[board=juno-r2/00:02:f7:00:67:93]:test_slack:                   passed    250/250 (100.0%)
            ThreeSmallTasks[board=juno-r2/00:02:f7:00:67:93]:test_task_placement:          undecided 1/250   (0.4%)
            TwoBigTasks[board=juno-r2/00:02:f7:00:67:93]:test_slack:                       passed    250/250 (100.0%)
            TwoBigTasks[board=juno-r2/00:02:f7:00:67:93]:test_task_placement:              undecided 9/250   (3.6%)
            TwoBigThreeSmall[board=juno-r2/00:02:f7:00:67:93]:test_slack:                  passed    250/250 (100.0%)
            TwoBigThreeSmall[board=juno-r2/00:02:f7:00:67:93]:test_task_placement:         undecided 5/250   (2.0%)
            
            Error: 0/14, Failed: 0/14, Undecided: 4/14, Skipped: 0/14, Passed: 10/14
        
        test/load_tracking (LISA-test) in 0:09:37 [BAD]
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=0@450000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=0@450000]:test_load_correctness:  passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=0@450000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=0@450000]:test_util_correctness:  passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=0@800000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=0@800000]:test_load_correctness:  passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=0@800000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=0@800000]:test_util_correctness:  passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=0@950000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=0@950000]:test_load_correctness:  FAILED 19/250  (7.6%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=0@950000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=0@950000]:test_util_correctness:  FAILED 19/250  (7.6%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=1@600000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=1@600000]:test_load_correctness:  FAILED 10/250  (4.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=1@600000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=1@600000]:test_util_correctness:  FAILED 10/250  (4.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=1@1000000]:test_load_behaviour:   passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=1@1000000]:test_load_correctness: FAILED 1/250   (0.4%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=1@1000000]:test_util_behaviour:   passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=1@1000000]:test_util_correctness: passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=1@1200000]:test_load_behaviour:   passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=1@1200000]:test_load_correctness: passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=1@1200000]:test_util_behaviour:   passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:iter_invariance_items[cpu=1@1200000]:test_util_correctness: passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:test_cpu_invariance:                                        passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:test_freq_invariance:                                       passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:test_load_behaviour:                                        passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:test_load_correctness:                                      FAILED 27/250  (10.8%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:test_util_behaviour:                                        passed 250/250 (100.0%)
            Invariance[board=juno-r2/00:02:f7:00:67:93]:test_util_correctness:                                      FAILED 27/250  (10.8%)
            NTasksCPUMigration[board=juno-r2/00:02:f7:00:67:93]:test_util_task_migration:                           passed 250/250 (100.0%)
            OneTaskCPUMigration[board=juno-r2/00:02:f7:00:67:93]:test_util_task_migration:                          FAILED 38/250  (15.2%)
            TwoTasksCPUMigration[board=juno-r2/00:02:f7:00:67:93]:test_util_task_migration:                         passed 250/250 (100.0%)
            
            Error: 0/33, Failed: 8/33, Undecided: 0/33, Skipped: 0/33, Passed: 25/33
        
        test/misfit (LISA-test) in 0:01:55 [GOOD]
            StaggeredFinishes[board=juno-r2/00:02:f7:00:67:93]:test_preempt_time: passed 250/250 (100.0%)
            StaggeredFinishes[board=juno-r2/00:02:f7:00:67:93]:test_throughput:   passed 250/250 (100.0%)
            
            Error: 0/2, Failed: 0/2, Undecided: 0/2, Skipped: 0/2, Passed: 2/2
        
        test/other (LISA-test) in 0:03:18 [GOOD]
            CapacitySanity[board=juno-r2/00:02:f7:00:67:93]:test_capacity_sanity:     passed 250/250 (100.0%)
            UserspaceSanity[board=juno-r2/00:02:f7:00:67:93]:test_performance_sanity: passed 250/250 (100.0%)
            
            Error: 0/2, Failed: 0/2, Undecided: 0/2, Skipped: 0/2, Passed: 2/2
        
        test/hotplug (LISA-test) in 0:01:45 [GOOD]
            HotplugTorture[board=juno-r2/00:02:f7:00:67:93]:test_cpus_alive:   passed 250/250 (100.0%)
            HotplugTorture[board=juno-r2/00:02:f7:00:67:93]:test_target_alive: passed 250/250 (100.0%)
            
            Error: 0/2, Failed: 0/2, Undecided: 0/2, Skipped: 0/2, Passed: 2/2
        
    (3) hikey960 (no test/hotplug, other stack (w/o linux-pm testing))
        
        testcase                                                                old%   new%  delta%       pvalue fix_iter# 
        -------------------------------------------------------------------------------------------------------------------
        Invariance:iter_invariance_items[cpu=0@1709000]:test_load_correctness   0.0%   5.2%    5.2%     2.09e-04        60 
        Invariance:iter_invariance_items[cpu=0@1709000]:test_util_correctness   0.0%   5.2%    5.2%     2.09e-04        60 
        Invariance:iter_invariance_items[cpu=0@1844000]:test_load_correctness   0.0%   4.8%    4.8%     4.28e-04        66 
        Invariance:iter_invariance_items[cpu=0@1844000]:test_util_correctness   0.0%   5.2%    5.2%     2.09e-04        60 
        Invariance:iter_invariance_items[cpu=0@533000]:test_load_correctness    4.1%   0.4%   -3.7%     5.51e-03           
        Invariance:iter_invariance_items[cpu=0@533000]:test_util_correctness    3.3%   0.4%   -2.9%     1.96e-02           
        Invariance:iter_invariance_items[cpu=4@2112000]:test_load_behaviour    50.4%   0.0%  -50.4%     1.02e-47           
        Invariance:iter_invariance_items[cpu=4@2112000]:test_util_behaviour    50.4%   0.0%  -50.4%     1.02e-47           
        Invariance:test_load_behaviour                                         50.2%   0.0%  -50.2%     2.05e-47           
        Invariance:test_load_correctness                                        4.1%  10.0%    5.9%     1.32e-02        46 
        Invariance:test_util_behaviour                                         50.2%   0.0%  -50.2%     2.05e-47           
        Invariance:test_util_correctness                                        3.3%  10.4%    7.1%     2.11e-03        27 
        OneSmallTask:test_task_placement                                       98.8%   1.6%  -97.2%    8.57e-134           
        StaggeredFinishes:test_throughput                                       3.7%   0.0%   -3.7%     1.62e-03           
        
        Description: Linux_version_5.5.0-rc1-gc2572976eb_(pdswauto@oss-arch-dev-jenkins-agent-04)_(gcc_version_4.9_20150123_(prerelease)_(GCC))__with_board_hikey960
        
        test/eas_behaviour (LISA-test) in 0:07:32 [BAD]
            EnergyModelWakeMigration[board=hikey960/00:0e:c6:ab:2b:1b]:test_slack:          passed 250/250 (100.0%)
            EnergyModelWakeMigration[board=hikey960/00:0e:c6:ab:2b:1b]:test_task_placement: passed 250/250 (100.0%)
            OneSmallTask[board=hikey960/00:0e:c6:ab:2b:1b]:test_slack:                      passed 250/250 (100.0%)
            OneSmallTask[board=hikey960/00:0e:c6:ab:2b:1b]:test_task_placement:             FAILED 4/250   (1.6%)
            RampDown[board=hikey960/00:0e:c6:ab:2b:1b]:test_slack:                          passed 250/250 (100.0%)
            RampDown[board=hikey960/00:0e:c6:ab:2b:1b]:test_task_placement:                 passed 250/250 (100.0%)
            RampUp[board=hikey960/00:0e:c6:ab:2b:1b]:test_slack:                            passed 250/250 (100.0%)
            RampUp[board=hikey960/00:0e:c6:ab:2b:1b]:test_task_placement:                   passed 250/250 (100.0%)
            ThreeSmallTasks[board=hikey960/00:0e:c6:ab:2b:1b]:test_slack:                   passed 250/250 (100.0%)
            ThreeSmallTasks[board=hikey960/00:0e:c6:ab:2b:1b]:test_task_placement:          passed 250/250 (100.0%)
            TwoBigTasks[board=hikey960/00:0e:c6:ab:2b:1b]:test_slack:                       passed 250/250 (100.0%)
            TwoBigTasks[board=hikey960/00:0e:c6:ab:2b:1b]:test_task_placement:              passed 250/250 (100.0%)
            TwoBigThreeSmall[board=hikey960/00:0e:c6:ab:2b:1b]:test_slack:                  passed 250/250 (100.0%)
            TwoBigThreeSmall[board=hikey960/00:0e:c6:ab:2b:1b]:test_task_placement:         passed 250/250 (100.0%)
            
            Error: 0/14, Failed: 1/14, Undecided: 0/14, Skipped: 0/14, Passed: 13/14
        
        test/load_tracking (LISA-test) in 0:16:05 [BAD]
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@533000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@533000]:test_load_correctness:  FAILED 1/250   (0.4%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@533000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@533000]:test_util_correctness:  FAILED 1/250   (0.4%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@999000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@999000]:test_load_correctness:  passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@999000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@999000]:test_util_correctness:  passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@1402000]:test_load_behaviour:   passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@1402000]:test_load_correctness: passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@1402000]:test_util_behaviour:   passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@1402000]:test_util_correctness: passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@1709000]:test_load_behaviour:   passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@1709000]:test_load_correctness: FAILED 13/250  (5.2%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@1709000]:test_util_behaviour:   passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@1709000]:test_util_correctness: FAILED 13/250  (5.2%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@1844000]:test_load_behaviour:   passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@1844000]:test_load_correctness: FAILED 12/250  (4.8%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@1844000]:test_util_behaviour:   passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=0@1844000]:test_util_correctness: FAILED 13/250  (5.2%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@903000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@903000]:test_load_correctness:  passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@903000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@903000]:test_util_correctness:  passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@1421000]:test_load_behaviour:   passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@1421000]:test_load_correctness: passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@1421000]:test_util_behaviour:   passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@1421000]:test_util_correctness: passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@1805000]:test_load_behaviour:   passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@1805000]:test_load_correctness: passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@1805000]:test_util_behaviour:   passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@1805000]:test_util_correctness: passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@2112000]:test_load_behaviour:   passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@2112000]:test_load_correctness: passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@2112000]:test_util_behaviour:   passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@2112000]:test_util_correctness: passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@2362000]:test_load_behaviour:   passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@2362000]:test_load_correctness: passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@2362000]:test_util_behaviour:   passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:iter_invariance_items[cpu=4@2362000]:test_util_correctness: passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:test_cpu_invariance:                                        passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:test_freq_invariance:                                       passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:test_load_behaviour:                                        passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:test_load_correctness:                                      FAILED 25/250  (10.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:test_util_behaviour:                                        passed 250/250 (100.0%)
            Invariance[board=hikey960/00:0e:c6:ab:2b:1b]:test_util_correctness:                                      FAILED 26/250  (10.4%)
            NTasksCPUMigration[board=hikey960/00:0e:c6:ab:2b:1b]:test_util_task_migration:                           passed 250/250 (100.0%)
            OneTaskCPUMigration[board=hikey960/00:0e:c6:ab:2b:1b]:test_util_task_migration:                          passed 250/250 (100.0%)
            TwoTasksCPUMigration[board=hikey960/00:0e:c6:ab:2b:1b]:test_util_task_migration:                         passed 250/250 (100.0%)
            
            Error: 0/49, Failed: 8/49, Undecided: 0/49, Skipped: 0/49, Passed: 41/49
        
        test/misfit (LISA-test) in 0:02:29 [GOOD]
            StaggeredFinishes[board=hikey960/00:0e:c6:ab:2b:1b]:test_preempt_time: passed 250/250 (100.0%)
            StaggeredFinishes[board=hikey960/00:0e:c6:ab:2b:1b]:test_throughput:   passed 250/250 (100.0%)
            
            Error: 0/2, Failed: 0/2, Undecided: 0/2, Skipped: 0/2, Passed: 2/2
        
        test/other (LISA-test) in 0:05:28 [GOOD]
            CapacitySanity[board=hikey960/00:0e:c6:ab:2b:1b]:test_capacity_sanity:     passed 250/250 (100.0%)
            UserspaceSanity[board=hikey960/00:0e:c6:ab:2b:1b]:test_performance_sanity: passed 250/250 (100.0%)
            
            Error: 0/2, Failed: 0/2, Undecided: 0/2, Skipped: 0/2, Passed: 2/2
        
    (4) pixel3:

        testcase                                                                old%   new%  delta%       pvalue fix_iter# 
        -------------------------------------------------------------------------------------------------------------------
        EnergyModelWakeMigration:test_slack                                    33.3% 100.0%   66.7%     2.23e-31         1 
        Invariance:iter_invariance_items[cpu=0@576000]:test_load_correctness    0.0%   7.6%    7.6%     5.14e-02        39 
        Invariance:iter_invariance_items[cpu=0@576000]:test_util_correctness    0.0%   8.0%    8.0%     5.28e-02        36 
        Invariance:iter_invariance_items[cpu=0@825600]:test_load_correctness   14.6%   0.0%  -14.6%     1.91e-06           
        Invariance:iter_invariance_items[cpu=0@825600]:test_util_correctness   14.6%   0.0%  -14.6%     1.91e-06           
        OneSmallTask:test_task_placement                                       54.2%   1.6%  -52.6%     3.14e-20           
        ThreeSmallTasks:test_task_placement                                    50.0%   0.4%  -49.6%     4.93e-21           
        TwoBigThreeSmall:test_task_placement                                   12.5%   0.0%  -12.5%     1.33e-05           
        UserspaceSanity:test_performance_sanity                                54.2%  77.2%   23.0%     2.03e-03         5 
        
        test/eas_behaviour (LISA-test) in 0:07:29 [BAD]
            EnergyModelWakeMigration[board=pixel3/3a:8b:c8:cd:97:f0]:test_slack:          FAILED    250/250 (100.0%)
            EnergyModelWakeMigration[board=pixel3/3a:8b:c8:cd:97:f0]:test_task_placement: passed    250/250 (100.0%)
            OneSmallTask[board=pixel3/3a:8b:c8:cd:97:f0]:test_slack:                      passed    250/250 (100.0%)
            OneSmallTask[board=pixel3/3a:8b:c8:cd:97:f0]:test_task_placement:             FAILED    4/250   (1.6%)
            RampDown[board=pixel3/3a:8b:c8:cd:97:f0]:test_slack:                          passed    250/250 (100.0%)
            RampDown[board=pixel3/3a:8b:c8:cd:97:f0]:test_task_placement:                 passed    250/250 (100.0%)
            RampUp[board=pixel3/3a:8b:c8:cd:97:f0]:test_slack:                            passed    250/250 (100.0%)
            RampUp[board=pixel3/3a:8b:c8:cd:97:f0]:test_task_placement:                   passed    250/250 (100.0%)
            ThreeSmallTasks[board=pixel3/3a:8b:c8:cd:97:f0]:test_slack:                   passed    250/250 (100.0%)
            ThreeSmallTasks[board=pixel3/3a:8b:c8:cd:97:f0]:test_task_placement:          FAILED    1/250   (0.4%)
            TwoBigTasks[board=pixel3/3a:8b:c8:cd:97:f0]:test_slack:                       FAILED    9/250   (3.6%)
            TwoBigTasks[board=pixel3/3a:8b:c8:cd:97:f0]:test_task_placement:              undecided 1/250   (0.4%)
            TwoBigThreeSmall[board=pixel3/3a:8b:c8:cd:97:f0]:test_slack:                  passed    250/250 (100.0%)
            TwoBigThreeSmall[board=pixel3/3a:8b:c8:cd:97:f0]:test_task_placement:         passed    250/250 (100.0%)
            
            Error: 0/14, Failed: 4/14, Undecided: 1/14, Skipped: 0/14, Passed: 9/14
        
        test/load_tracking (LISA-test) in 0:30:18 [BAD]
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@300000]:test_load_behaviour:    undecided 1/250   (0.4%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@300000]:test_load_correctness:  FAILED    50/250  (20.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@300000]:test_util_behaviour:    undecided 1/250   (0.4%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@300000]:test_util_correctness:  FAILED    15/250  (6.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@576000]:test_load_behaviour:    passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@576000]:test_load_correctness:  FAILED    19/250  (7.6%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@576000]:test_util_behaviour:    passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@576000]:test_util_correctness:  FAILED    20/250  (8.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@825600]:test_load_behaviour:    passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@825600]:test_load_correctness:  passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@825600]:test_util_behaviour:    passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@825600]:test_util_correctness:  passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1056000]:test_load_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1056000]:test_load_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1056000]:test_util_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1056000]:test_util_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1324800]:test_load_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1324800]:test_load_correctness: FAILED    11/250  (4.4%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1324800]:test_util_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1324800]:test_util_correctness: FAILED    13/250  (5.2%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1516800]:test_load_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1516800]:test_load_correctness: FAILED    3/250   (1.2%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1516800]:test_util_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1516800]:test_util_correctness: FAILED    3/250   (1.2%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1689600]:test_load_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1689600]:test_load_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1689600]:test_util_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1689600]:test_util_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1766400]:test_load_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1766400]:test_load_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1766400]:test_util_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=0@1766400]:test_util_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@825600]:test_load_behaviour:    passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@825600]:test_load_correctness:  passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@825600]:test_util_behaviour:    passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@825600]:test_util_correctness:  passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@1056000]:test_load_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@1056000]:test_load_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@1056000]:test_util_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@1056000]:test_util_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@1459200]:test_load_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@1459200]:test_load_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@1459200]:test_util_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@1459200]:test_util_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@1766400]:test_load_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@1766400]:test_load_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@1766400]:test_util_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@1766400]:test_util_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@1996800]:test_load_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@1996800]:test_load_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@1996800]:test_util_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@1996800]:test_util_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@2323200]:test_load_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@2323200]:test_load_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@2323200]:test_util_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@2323200]:test_util_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@2553600]:test_load_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@2553600]:test_load_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@2553600]:test_util_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@2553600]:test_util_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@2649600]:test_load_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@2649600]:test_load_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@2649600]:test_util_behaviour:   passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:iter_invariance_items[cpu=4@2649600]:test_util_correctness: passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:test_cpu_invariance:                                        passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:test_freq_invariance:                                       passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:test_load_behaviour:                                        passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:test_load_correctness:                                      FAILED    73/250  (29.2%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:test_util_behaviour:                                        passed    250/250 (100.0%)
            Invariance[board=pixel3/3a:8b:c8:cd:97:f0]:test_util_correctness:                                      FAILED    47/250  (18.8%)
            NTasksCPUMigration[board=pixel3/3a:8b:c8:cd:97:f0]:test_util_task_migration:                           FAILED    250/250 (100.0%)
            OneTaskCPUMigration[board=pixel3/3a:8b:c8:cd:97:f0]:test_util_task_migration:                          FAILED    13/250  (5.2%)
            TwoTasksCPUMigration[board=pixel3/3a:8b:c8:cd:97:f0]:test_util_task_migration:                         passed    250/250 (100.0%)
            
            Error: 0/73, Failed: 12/73, Undecided: 2/73, Skipped: 0/73, Passed: 59/73
        
        test/misfit (LISA-test) in 0:02:05 [GOOD]
            StaggeredFinishes[board=pixel3/3a:8b:c8:cd:97:f0]:test_preempt_time: passed 250/250 (100.0%)
            StaggeredFinishes[board=pixel3/3a:8b:c8:cd:97:f0]:test_throughput:   passed 250/250 (100.0%)
            
            Error: 0/2, Failed: 0/2, Undecided: 0/2, Skipped: 0/2, Passed: 2/2
        
        test/other (LISA-test) in 0:04:27 [BAD]
            CapacitySanity[board=pixel3/3a:8b:c8:cd:97:f0]:test_capacity_sanity:     passed 250/250 (100.0%)
            UserspaceSanity[board=pixel3/3a:8b:c8:cd:97:f0]:test_performance_sanity: FAILED 193/250 (77.2%)
            
            Error: 0/2, Failed: 1/2, Undecided: 0/2, Skipped: 0/2, Passed: 1/2
        
        test/hotplug (LISA-test) in 0:01:49 [GOOD]
            HotplugTorture[board=pixel3/3a:8b:c8:cd:97:f0]:test_cpus_alive:   passed 250/250 (100.0%)
            HotplugTorture[board=pixel3/3a:8b:c8:cd:97:f0]:test_target_alive: passed 250/250 (100.0%)
            
            Error: 0/2, Failed: 0/2, Undecided: 0/2, Skipped: 0/2, Passed: 2/2
        
    (5) tc2 (no test/hotplug):
        
        testcase                                       old%   new%  delta%       pvalue fix_iter# 
        ------------------------------------------------------------------------------------------
        OneSmallTask:test_task_placement              96.4%   0.0%  -96.4%    2.15e-133           
        ThreeSmallTasks:test_task_placement           18.4%   0.0%  -18.4%     2.90e-15           
        TwoTasksCPUMigration:test_util_task_migration 56.8%   0.0%  -56.8%     1.23e-55           

        Description: Linux_version_5.5.0-rc1-gd394df8202_(pdswauto@oss-arch-dev-jenkins-agent-04)_(gcc_version_8.3.0_(GNU_Toolchain_for_the_A-profile_Architecture_8.3-2019.03_(arm-rel-8.36)))__with_board_tc2
        
        test/eas_behaviour (LISA-test) in 0:07:29 [BAD]
            EnergyModelWakeMigration[board=tc2/00:02:F7:00:48:A4]:test_slack:          passed 250/250 (100.0%)
            EnergyModelWakeMigration[board=tc2/00:02:F7:00:48:A4]:test_task_placement: passed 250/250 (100.0%)
            OneSmallTask[board=tc2/00:02:F7:00:48:A4]:test_slack:                      passed 250/250 (100.0%)
            OneSmallTask[board=tc2/00:02:F7:00:48:A4]:test_task_placement:             passed 250/250 (100.0%)
            RampDown[board=tc2/00:02:F7:00:48:A4]:test_slack:                          FAILED 1/250   (0.4%)
            RampDown[board=tc2/00:02:F7:00:48:A4]:test_task_placement:                 passed 250/250 (100.0%)
            RampUp[board=tc2/00:02:F7:00:48:A4]:test_slack:                            passed 250/250 (100.0%)
            RampUp[board=tc2/00:02:F7:00:48:A4]:test_task_placement:                   passed 250/250 (100.0%)
            ThreeSmallTasks[board=tc2/00:02:F7:00:48:A4]:test_slack:                   passed 250/250 (100.0%)
            ThreeSmallTasks[board=tc2/00:02:F7:00:48:A4]:test_task_placement:          passed 250/250 (100.0%)
            TwoBigTasks[board=tc2/00:02:F7:00:48:A4]:test_slack:                       passed 250/250 (100.0%)
            TwoBigTasks[board=tc2/00:02:F7:00:48:A4]:test_task_placement:              passed 250/250 (100.0%)
            TwoBigThreeSmall[board=tc2/00:02:F7:00:48:A4]:test_slack:                  passed 250/250 (100.0%)
            TwoBigThreeSmall[board=tc2/00:02:F7:00:48:A4]:test_task_placement:         FAILED 2/250   (0.8%)
            
            Error: 0/14, Failed: 2/14, Undecided: 0/14, Skipped: 0/14, Passed: 12/14
            
        test/load_tracking (LISA-test) in 0:23:12 [BAD]
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@350000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@350000]:test_load_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@350000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@350000]:test_util_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@400000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@400000]:test_load_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@400000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@400000]:test_util_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@500000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@500000]:test_load_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@500000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@500000]:test_util_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@600000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@600000]:test_load_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@600000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@600000]:test_util_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@700000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@700000]:test_load_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@700000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@700000]:test_util_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@800000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@800000]:test_load_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@800000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@800000]:test_util_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@900000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@900000]:test_load_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@900000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@900000]:test_util_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@1000000]:test_load_behaviour:   passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@1000000]:test_load_correctness: passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@1000000]:test_util_behaviour:   passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=0@1000000]:test_util_correctness: passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@500000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@500000]:test_load_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@500000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@500000]:test_util_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@600000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@600000]:test_load_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@600000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@600000]:test_util_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@700000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@700000]:test_load_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@700000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@700000]:test_util_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@800000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@800000]:test_load_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@800000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@800000]:test_util_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@900000]:test_load_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@900000]:test_load_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@900000]:test_util_behaviour:    passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@900000]:test_util_correctness:  passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@1000000]:test_load_behaviour:   passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@1000000]:test_load_correctness: passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@1000000]:test_util_behaviour:   passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@1000000]:test_util_correctness: passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@1100000]:test_load_behaviour:   passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@1100000]:test_load_correctness: passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@1100000]:test_util_behaviour:   passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@1100000]:test_util_correctness: passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@1200000]:test_load_behaviour:   passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@1200000]:test_load_correctness: passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@1200000]:test_util_behaviour:   passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:iter_invariance_items[cpu=1@1200000]:test_util_correctness: passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:test_cpu_invariance:                                        passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:test_freq_invariance:                                       passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:test_load_behaviour:                                        passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:test_load_correctness:                                      passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:test_util_behaviour:                                        passed 250/250 (100.0%)
            Invariance[board=tc2/00:02:F7:00:48:95]:test_util_correctness:                                      passed 250/250 (100.0%)
            NTasksCPUMigration[board=tc2/00:02:F7:00:48:95]:test_util_task_migration:                           passed 250/250 (100.0%)
            OneTaskCPUMigration[board=tc2/00:02:F7:00:48:95]:test_util_task_migration:                          FAILED 250/250 (100.0%)
            TwoTasksCPUMigration[board=tc2/00:02:F7:00:48:95]:test_util_task_migration:                         passed 250/250 (100.0%)
            
            Error: 0/73, Failed: 1/73, Undecided: 0/73, Skipped: 0/73, Passed: 72/73

        test/misfit (LISA-test) in 0:02:08 [BAD]
            StaggeredFinishes[board=tc2/00:02:F7:00:48:A4]:test_preempt_time: passed 250/250 (100.0%)
            StaggeredFinishes[board=tc2/00:02:F7:00:48:A4]:test_throughput:   FAILED 11/250  (4.4%)
            
            Error: 0/2, Failed: 1/2, Undecided: 0/2, Skipped: 0/2, Passed: 1/2
        
        test/other (LISA-test) in 0:05:50 [GOOD]
            CapacitySanity[board=tc2/00:02:F7:00:48:A4]:test_capacity_sanity:     passed 250/250 (100.0%)
            UserspaceSanity[board=tc2/00:02:F7:00:48:A4]:test_performance_sanity: passed 250/250 (100.0%)
            
            Error: 0/2, Failed: 0/2, Undecided: 0/2, Skipped: 0/2, Passed: 2/2