EAS Development for Mainline Linux

EAS development for mainline Linux takes place on LKML and Linaro eas-dev mailing lists where proposed patches are discussed with the community. Arm now provides a pre-integrated version of latest EAS patches for mainline, for upstream development use.

EAS mainline is a moving integration branch tracking tip sched/core, mainline release candidate or linux-pm testing and adding the latest versions of EAS related patch sets targeted at the mainline kernel. The patches may already be under discussion on relevant open source mailing lists or waiting here while their dependencies get resolved and merged upstream.

The integration branch comes as is with synthetic test case testing only. The intention is to have a single branch containing the full EAS picture for mainline Linux kernel developers interested in tracking progress. This EAS mainline integration branch is not suitable for product development. For product, please use the AOSP common kernel.

Source code

EAS mainline integration branches are made available regularly on:

        git://linux-arm.org/linux-power.git

The releases are tagged with the date e.g. -20190310 so old integrations are available for comparison

Latest: integration-20190820

Tree: eas/next/integration

Base: eas/next/integration-base

New changes:

  1. New patch 'sched/fair: Reduce complexity of energy calculation'  in em_overutil_and_eas topic

Features:

  1. Util(ization) clamping (remaining cgroup interface) (v14 on lkml)
  2. Intellegent Power Allocation (IPA) as EM (Energy Model) user (v7 on lkml)
  3. Schedutil as EM user (v2 on lkml)
  4. Util(ization) update during Idle migration (v1 on lkml)

Features in Linux v5.3-rc1:

  1. Tracepoints (PELT and over-utilzation) v3 patch-set
  2. Util(ization) clamping (core and per-task interface) v10 patch-set

Features in Linux mainline:

  1. Frequency Invariant Engine (FIE) and CPU Invariant Engine (v4.13/v4.15)
  2. Idle CPU PELT (Per-Entity Load-Tracking) update (v4.17)
  3. Util(ization) Est(imated) (v4.17)
  4. Misfit task, i.e. forcing migration of running tasks that do not fit on the CPU they are currently running on (v4.20)
  5. Runtime scheduler domain flag detection (v4.20)
  6. Per-cpu Energy Model and Energy Aware Scheduling (v5.0)

Platforms we test on:

  1. Arm Juno r0 (arm64)
  2. Arm Juno r2 (arm64)
  3. Hikey960 (arm64) (needs out-of-tree patches  (hikey960_mainline topic))
  4. Arm TC2 (arm32) (needs out-of-tree patch (TC2_mainline topic))
  5. Pixel3 (arm64) (needs out-of-tree patch (pixel3_mainline topic))

Next work items:

  1. Pushing Features upstream
  2. 100% test pass rate on all platforms
  • Test Results (integration 20190820) 

       juno-r0:

       test/eas_behaviour (LISA-test) [GOOD]
            EnergyModelWakeMigration:test_slack:          passed    250/250 (100.0%)
            EnergyModelWakeMigration:test_task_placement: passed    250/250 (100.0%)
            OneSmallTask:test_slack:                      passed    250/250 (100.0%)
            OneSmallTask:test_task_placement:             passed    250/250 (100.0%)
            RampDown:test_slack:                          passed    250/250 (100.0%)
            RampDown:test_task_placement:                 passed    250/250 (100.0%)
            RampUp:test_slack:                            passed    250/250 (100.0%)
            RampUp:test_task_placement:                   passed    250/250 (100.0%)
            ThreeSmallTasks:test_slack:                   passed    250/250 (100.0%)
            ThreeSmallTasks:test_task_placement:          passed    250/250 (100.0%)
            TwoBigTasks:test_slack:                       passed    250/250 (100.0%)
            TwoBigTasks:test_task_placement:              passed    250/250 (100.0%)
            TwoBigThreeSmall:test_slack:                  passed    250/250 (100.0%)
            TwoBigThreeSmall:test_task_placement:         undecided 2/250   (0.8%)

            Error: 0/14, Failed: 0/14, Undecided: 1/14, Skipped: 0/14, Passed: 13/14

        test/load_tracking (LISA-test) [BAD]
            Invariance:iter_invariance_items[cpu=0@450000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@450000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@575000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@575000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@700000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@700000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@775000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@775000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@850000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@850000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@450000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@450000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@625000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@625000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@800000]:test_task_load_avg:  FAILED 1/250   (0.4%)
            Invariance:iter_invariance_items[cpu=1@800000]:test_task_util_avg:  FAILED 1/250   (0.4%)
            Invariance:iter_invariance_items[cpu=1@950000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@950000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@1100000]:test_task_load_avg: passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@1100000]:test_task_util_avg: passed 250/250 (100.0%)
            Invariance:test_cpu_invariance:                                     passed 250/250 (100.0%)
            Invariance:test_freq_invariance:                                    passed 250/250 (100.0%)
            Invariance:test_task_load_avg:                                      FAILED 1/250   (0.4%)
            Invariance:test_task_util_avg:                                      FAILED 1/250   (0.4%)
            NTasksCPUMigration:test_util_task_migration:                        FAILED 1/250   (0.4%)
            OneTaskCPUMigration:test_util_task_migration:                       passed 250/250 (100.0%)
            PELTTask:test_load_avg_behaviour:                                   passed 250/250 (100.0%)
            PELTTask:test_load_avg_range:                                       FAILED 32/250  (12.8%)
            PELTTask:test_util_avg_behaviour:                                   passed 250/250 (100.0%)
            PELTTask:test_util_avg_range:                                       FAILED 32/250  (12.8%)
            TwoTasksCPUMigration:test_util_task_migration:                      passed 250/250 (100.0%)

            Error: 0/31, Failed: 7/31, Undecided: 0/31, Skipped: 0/31, Passed: 24/31

        test/misfit (LISA-test) [BAD]
            StaggeredFinishes:test_preempt_time: FAILED 1/250   (0.4%)
            StaggeredFinishes:test_throughput:   passed 250/250 (100.0%)

            Error: 0/2, Failed: 1/2, Undecided: 0/2, Skipped: 0/2, Passed: 1/2 

        test/other (LISA-test) [GOOD]
            CapacitySanity:test_capacity_sanity:     passed 250/250 (100.0%)
            UserspaceSanity:test_performance_sanity: passed 250/250 (100.0%)

            Error: 0/2, Failed: 0/2, Undecided: 0/2, Skipped: 0/2, Passed: 2/2 

        test/hotplug (LISA-test) [GOOD]
            HotplugTorture:test_cpus_alive:   passed 250/250 (100.0%)
            HotplugTorture:test_target_alive: passed 250/250 (100.0%)

            Error: 0/2, Failed: 0/2, Undecided: 0/2, Skipped: 0/2, Passed: 2/2 

        juno-r2:

        test/eas_behaviour (LISA-test) [BAD]
            EnergyModelWakeMigration:test_slack:          passed    250/250 (100.0%)
            EnergyModelWakeMigration:test_task_placement: passed    250/250 (100.0%)
            OneSmallTask:test_slack:                      passed    250/250 (100.0%)
            OneSmallTask:test_task_placement:             FAILED    1/250   (0.4%)
            RampDown:test_slack:                          passed    250/250 (100.0%)
            RampDown:test_task_placement:                 passed    250/250 (100.0%)
            RampUp:test_slack:                            passed    250/250 (100.0%)
            RampUp:test_task_placement:                   passed    250/250 (100.0%)
            ThreeSmallTasks:test_slack:                   passed    250/250 (100.0%)
            ThreeSmallTasks:test_task_placement:          passed    250/250 (100.0%)
            TwoBigTasks:test_slack:                       FAILED    2/250   (0.8%)
            TwoBigTasks:test_task_placement:              undecided 6/250   (2.4%)
            TwoBigThreeSmall:test_slack:                  FAILED    2/250   (0.8%)
            TwoBigThreeSmall:test_task_placement:         undecided 6/250   (2.4%)

            Error: 0/14, Failed: 3/14, Undecided: 2/14, Skipped: 0/14, Passed: 9/14

        test/load_tracking (LISA-test) [BAD]
            Invariance:iter_invariance_items[cpu=0@450000]:test_task_load_avg:  passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@450000]:test_task_util_avg:  passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@800000]:test_task_load_avg:  passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@800000]:test_task_util_avg:  passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@950000]:test_task_load_avg:  passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@950000]:test_task_util_avg:  passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@600000]:test_task_load_avg:  passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@600000]:test_task_util_avg:  passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@1000000]:test_task_load_avg: passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@1000000]:test_task_util_avg: passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@1200000]:test_task_load_avg: passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@1200000]:test_task_util_avg: passed    250/250 (100.0%)
            Invariance:test_cpu_invariance:                                     passed    250/250 (100.0%)
            Invariance:test_freq_invariance:                                    passed    250/250 (100.0%)
            Invariance:test_task_load_avg:                                      passed    250/250 (100.0%)
            Invariance:test_task_util_avg:                                      passed    250/250 (100.0%)
            NTasksCPUMigration:test_util_task_migration:                        FAILED    250/250 (100.0%)
            OneTaskCPUMigration:test_util_task_migration:                       FAILED    3/250   (1.2%)
            PELTTask:test_load_avg_behaviour:                                   passed    250/250 (100.0%)
            PELTTask:test_load_avg_range:                                       passed    250/250 (100.0%)
            PELTTask:test_util_avg_behaviour:                                   passed    250/250 (100.0%)
            PELTTask:test_util_avg_range:                                       passed    250/250 (100.0%)
            TwoTasksCPUMigration:test_util_task_migration:                      undecided 1/250   (0.4%)

            Error: 0/23, Failed: 2/23, Undecided: 1/23, Skipped: 0/23, Passed: 20/23

        test/misfit (LISA-test) [BAD]
            StaggeredFinishes:test_preempt_time: passed    249/249 (100.0%)
            StaggeredFinishes:test_throughput:   undecided 2/249   (0.8%)

            Error: 0/2, Failed: 0/2, Undecided: 1/2, Skipped: 0/2, Passed: 1/2 

        test/other (LISA-test) [GOOD]
            CapacitySanity:test_capacity_sanity:     passed 250/250 (100.0%)
            UserspaceSanity:test_performance_sanity: passed 250/250 (100.0%)

            Error: 0/2, Failed: 0/2, Undecided: 0/2, Skipped: 0/2, Passed: 2/2 

        test/hotplug (LISA-test) [GOOD]
            HotplugTorture:test_cpus_alive:   passed 250/250 (100.0%)
            HotplugTorture:test_target_alive: passed 250/250 (100.0%)

            Error: 0/2, Failed: 0/2, Undecided: 0/2, Skipped: 0/2, Passed: 2/2 

        hikey960:

        test/eas_behaviour (LISA-test) [BAD]
            EnergyModelWakeMigration:test_slack:          passed 250/250 (100.0%)
            EnergyModelWakeMigration:test_task_placement: passed 250/250 (100.0%)
            OneSmallTask:test_slack:                      passed 250/250 (100.0%)
            OneSmallTask:test_task_placement:             passed 250/250 (100.0%)
            RampDown:test_slack:                          passed 250/250 (100.0%)
            RampDown:test_task_placement:                 passed 250/250 (100.0%)
            RampUp:test_slack:                            passed 250/250 (100.0%)
            RampUp:test_task_placement:                   passed 250/250 (100.0%)
            ThreeSmallTasks:test_slack:                   passed 250/250 (100.0%)
            ThreeSmallTasks:test_task_placement:          passed 250/250 (100.0%)
            TwoBigTasks:test_slack:                       passed 250/250 (100.0%)
            TwoBigTasks:test_task_placement:              passed 250/250 (100.0%)
            TwoBigThreeSmall:test_slack:                  passed 250/250 (100.0%)
            TwoBigThreeSmall:test_task_placement:         FAILED 2/250   (0.8%)

            Error: 0/14, Failed: 1/14, Undecided: 0/14, Skipped: 0/14, Passed: 13/14

        test/load_tracking (LISA-test) [GOOD]
            Invariance:iter_invariance_items[cpu=0@533000]:test_task_load_avg:  undecided 1/250   (0.4%)
            Invariance:iter_invariance_items[cpu=0@533000]:test_task_util_avg:  undecided 1/250   (0.4%)
            Invariance:iter_invariance_items[cpu=0@999000]:test_task_load_avg:  passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@999000]:test_task_util_avg:  passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@1402000]:test_task_load_avg: passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@1402000]:test_task_util_avg: passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@1709000]:test_task_load_avg: passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@1709000]:test_task_util_avg: passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@1844000]:test_task_load_avg: passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@1844000]:test_task_util_avg: passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=4@903000]:test_task_load_avg:  passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=4@903000]:test_task_util_avg:  passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=4@1421000]:test_task_load_avg: passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=4@1421000]:test_task_util_avg: passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=4@1805000]:test_task_load_avg: passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=4@1805000]:test_task_util_avg: passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=4@2112000]:test_task_load_avg: passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=4@2112000]:test_task_util_avg: passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=4@2362000]:test_task_load_avg: passed    250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=4@2362000]:test_task_util_avg: passed    250/250 (100.0%)
            Invariance:test_cpu_invariance:                                     passed    250/250 (100.0%)
            Invariance:test_freq_invariance:                                    passed    250/250 (100.0%)
            Invariance:test_task_load_avg:                                      passed    250/250 (100.0%)
            Invariance:test_task_util_avg:                                      passed    250/250 (100.0%)
            NTasksCPUMigration:test_util_task_migration:                        passed    250/250 (100.0%)
            OneTaskCPUMigration:test_util_task_migration:                       passed    250/250 (100.0%)
            PELTTask:test_load_avg_behaviour:                                   passed    250/250 (100.0%)
            PELTTask:test_load_avg_range:                                       passed    250/250 (100.0%)
            PELTTask:test_util_avg_behaviour:                                   passed    250/250 (100.0%)
            PELTTask:test_util_avg_range:                                       passed    250/250 (100.0%)
            TwoTasksCPUMigration:test_util_task_migration:                      passed    250/250 (100.0%)

            Error: 0/31, Failed: 0/31, Undecided: 2/31, Skipped: 0/31, Passed: 29/31

        test/misfit (LISA-test) [BAD]
            StaggeredFinishes:test_preempt_time: FAILED 3/250   (1.2%)
            StaggeredFinishes:test_throughput:   passed 250/250 (100.0%)

            Error: 0/2, Failed: 1/2, Undecided: 0/2, Skipped: 0/2, Passed: 1/2 

        test/other (LISA-test) [GOOD]
            CapacitySanity:test_capacity_sanity:     passed 250/250 (100.0%)
            UserspaceSanity:test_performance_sanity: passed 250/250 (100.0%)

            Error: 0/2, Failed: 0/2, Undecided: 0/2, Skipped: 0/2, Passed: 2/2 

        TC2:

        test/eas_behaviour (LISA-test) [GOOD]
            EnergyModelWakeMigration:test_slack:          passed 250/250 (100.0%)
            EnergyModelWakeMigration:test_task_placement: passed 250/250 (100.0%)
            OneSmallTask:test_slack:                      passed 250/250 (100.0%)
            OneSmallTask:test_task_placement:             passed 250/250 (100.0%)
            RampDown:test_slack:                          passed 250/250 (100.0%)
            RampDown:test_task_placement:                 passed 250/250 (100.0%)
            RampUp:test_slack:                            passed 250/250 (100.0%)
            RampUp:test_task_placement:                   passed 250/250 (100.0%)
            ThreeSmallTasks:test_slack:                   passed 250/250 (100.0%)
            ThreeSmallTasks:test_task_placement:          passed 250/250 (100.0%)
            TwoBigTasks:test_slack:                       passed 250/250 (100.0%)
            TwoBigTasks:test_task_placement:              passed 250/250 (100.0%)
            TwoBigThreeSmall:test_slack:                  passed 250/250 (100.0%)
            TwoBigThreeSmall:test_task_placement:         passed 250/250 (100.0%)

            Error: 0/14, Failed: 0/14, Undecided: 0/14, Skipped: 0/14, Passed: 14/14

        test/load_tracking (LISA-test) [BAD]
            Invariance:iter_invariance_items[cpu=0@350000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@350000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@400000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@400000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@500000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@500000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@600000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@600000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@700000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@700000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@800000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@800000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@900000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@900000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@1000000]:test_task_load_avg: passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=0@1000000]:test_task_util_avg: passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@500000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@500000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@600000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@600000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@700000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@700000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@800000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@800000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@900000]:test_task_load_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@900000]:test_task_util_avg:  passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@1000000]:test_task_load_avg: passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@1000000]:test_task_util_avg: passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@1100000]:test_task_load_avg: passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@1100000]:test_task_util_avg: passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@1200000]:test_task_load_avg: passed 250/250 (100.0%)
            Invariance:iter_invariance_items[cpu=1@1200000]:test_task_util_avg: passed 250/250 (100.0%)
            Invariance:test_cpu_invariance:                                     passed 250/250 (100.0%)
            Invariance:test_freq_invariance:                                    passed 250/250 (100.0%)
            Invariance:test_task_load_avg:                                      passed 250/250 (100.0%)
            Invariance:test_task_util_avg:                                      passed 250/250 (100.0%)
            NTasksCPUMigration:test_util_task_migration:                        passed 250/250 (100.0%)
            OneTaskCPUMigration:test_util_task_migration:                       FAILED 250/250 (100.0%)
            PELTTask:test_load_avg_behaviour:                                   passed 250/250 (100.0%)
            PELTTask:test_load_avg_range:                                       passed 250/250 (100.0%)
            PELTTask:test_util_avg_behaviour:                                   passed 250/250 (100.0%)
            PELTTask:test_util_avg_range:                                       passed 250/250 (100.0%)
            TwoTasksCPUMigration:test_util_task_migration:                      passed 250/250 (100.0%)

            Error: 0/43, Failed: 1/43, Undecided: 0/43, Skipped: 0/43, Passed: 42/43

        test/misfit (LISA-test) [BAD]
            StaggeredFinishes:test_preempt_time: passed 250/250 (100.0%)
            StaggeredFinishes:test_throughput:   FAILED 32/250  (12.8%)

            Error: 0/2, Failed: 1/2, Undecided: 0/2, Skipped: 0/2, Passed: 1/2

        test/other (LISA-test) [GOOD]
            CapacitySanity:test_capacity_sanity:     passed 250/250 (100.0%)
            UserspaceSanity:test_performance_sanity: passed 250/250 (100.0%)

            Error: 0/2, Failed: 0/2, Undecided: 0/2, Skipped: 0/2, Passed: 2/2

        test/hotplug (LISA-test) [GOOD]
            HotplugTorture:test_cpus_alive:   passed 250/250 (100.0%)
            HotplugTorture:test_target_alive: passed 250/250 (100.0%)

            Error: 0/2, Failed: 0/2, Undecided: 0/2, Skipped: 0/2, Passed: 2/2