Add reference workload support

A backend is an abstraction that maps the layers of a network graph to the hardware that executes the supported layers. Backends create specific workloads for the layers that they support. A workload includes the implementation of the layer and is used to queue a layer for computation. Each layer is executed using a workload.

The CpuRef reference backend represents the non-accelerated CPU of the device Arm NN runs on. The CpuRef backend is mainly used for testing purposes. Reference backend workloads are stored under backends/reference/workloads.

  1. Add a workload header and source file for your layer to src/backends/<BackendName>/workloads/<BackendName><LayerName>Workload.hpp and src/backends/<BackendName>/workloads/<BackendName><LayerName>Workload.cpp.

    For example, to implement src/backends/reference/workloads/RefSoftmaxWorkload.hpp and src/backends/reference/workloads/RefSoftmaxWorkload.cpp for an example SoftmaxLayer, make the following changes in /<BackendName>/<LayerName>Workload.hpp:

    #pragma once
    
    #include <backendsCommon/Workload.hpp>
    #include <backendsCommon/WorkloadData.hpp>
    
    namespace armnn
    {
    
    class RefSoftmaxWorkload : public BaseWorkload<SoftmaxQueueDescriptor>
    {
    public:
        using BaseWorkload<SoftmaxQueueDescriptor>::BaseWorkload;
        virtual void Execute() const override;
    };
    
    } //namespace armnn

    The following code shows the example changes to implement src/backends/reference/workloads/ RefSoftmaxWorkload.hpp and src/backends/reference/workloads/ RefSoftmaxWorkload.cpp in /<BackendName><LayerName>Workload.cpp.

    #include "RefSoftmaxWorkload.hpp"
    
    ...
    
    namespace armnn
    {
    
    void RefSoftmaxWorkload::Execute() const
    {
        // Execute Implementation
    }
    } //namespace armnn
  2. Add a Create<LayerName>() function to the workload factory. The <LayerName>LayerCreateWorkload() function calls the Create<LayerName>() function as part of the workload creation process. For example, the following code adds the CreateSoftmax() function to RefWorkloadFactory.hpp:
    std::unique_ptr<IWorkload> RefWorkloadFactory::CreateSoftmax(const SoftmaxQueueDescriptor& descriptor,
                                                                 const WorkloadInfo& info) const
    {
        return std::make_unique<RefSoftmaxWorkload>(descriptor, info);
    }
  3. Add your workload to backends/reference/workloads/CMakeLists.txt. The following code shows how to add an example SoftmaxLayer workload:
    RefSliceWorkload.cpp
        RefSliceWorkload.hpp
        RefSoftmaxWorkload.cpp
        RefSoftmaxWorkload.hpp
        RefSpaceToBatchNdWorkload.cpp
        RefSpaceToBatchNdWorkload.hpp
    

    Add your workload to backends/reference/backend.mk for the Android NN HAL driver.  You must add your workload to backends/reference/backend.mk. The Android NN HAL driver requires this workload even if you do not plan full Android NN support for the operator. See the Add Android NN support section for more information. The following code shows the addition of a workload for an example SoftmaxLayer:

    workloads/RefSliceWorkload.cpp \
            workloads/RefSoftmaxWorkload.cpp \
            workloads/RefSpaceToBatchNdWorkload.cpp \
  4. Add your workload to a list in backends/reference/workloads/RefWorkloads.hpp. The following code shows the addition of the workload for an example SoftmaxLayer:
    #include "Resize.hpp"
    #include "Softmax.hpp"
    #include "Splitter.hpp"
    
  5. Add the Is<LayerName>Supported() function to backends/reference/RefLayerSupport.hpp and backends/reference/RefLayerSupport.cpp. The calling of this function does not depend on whether the related backend supports the operator <LayerName>. Ensure that you add the supported conditions inside the function. The following code shows this addition to the RefLayerSupport.hpp file:
    bool IsSoftmaxSupported(const TensorInfo& input,
                                const TensorInfo& output,
                                const SoftmaxDescriptor& descriptor,
                                Optional<std::string&> reasonIfUnsupported = EmptyOptional()) const override;
    

    The following code shows this addition to the RefLayerSupport.cpp file:

    bool RefLayerSupport::IsSoftmaxSupported(const TensorInfo& input,
                                             const TensorInfo& output,
                                             const SoftmaxDescriptor& descriptor,
                                             Optional<std::string&> reasonIfUnsupported) const
    {
        IgnoreUnused(descriptor);
        bool supported = true;
        std::array<DataType,7> supportedTypes =
        {
            DataType::BFloat16,
            DataType::Float32,
            DataType::Float16,
            DataType::QSymmS8,
            DataType::QAsymmS8,
            DataType::QAsymmU8,
            DataType::QSymmS16
        };
    
        supported &= CheckSupportRule(TypeAnyOf(input, supportedTypes), reasonIfUnsupported,
                                      "Reference Softmax: output type not supported");
    
        supported &= CheckSupportRule(TypeAnyOf(output, supportedTypes), reasonIfUnsupported,
                                      "Reference Softmax: input type not supported");
    
        supported &= CheckSupportRule(TypesAreEqual(input, output), reasonIfUnsupported,
                                      "Reference Softmax: input type not supported");
    
        return supported;
    }

Add layer unit tests

Layer unit tests use the most basic graph possible to check the functionality of each new layer or new operator with different data types and configurations. To do this check, a layer unit test uses a basic graph like InputLayer→NewLayer→OutputLayer. You can write variations of the unit test to use different-sized input tensors and different quantization parameters, configuration parameters, and data types with the basic graph.

To verify the functionality of the new layer or operator, create a unit test suite.

To create a unit test suite:

  1. Add the header file, <LayerName>TestImpl.hpp, and source file, <LayerName>TestImpl.cpp, to backends/backendsCommon/test/LayerTests. <LayerName> is the name of the layer you are implementing.
  2. Declare one or more tests for your new layer in the header file. The following code shows an example unit test for the SoftmaxLayer:
  3. #pragma once
    
    #include "LayerTestResult.hpp"
    
    #include <Half.hpp>
    
    #include <armnn/backends/IBackendInternal.hpp>
    #include <backendsCommon/WorkloadFactory.hpp>
    
    LayerTestResult<float, 2> SimpleSoftmaxTest(
        armnn::IWorkloadFactory& workloadFactory,
        const armnn::IBackendInternal::IMemoryManagerSharedPtr& memoryManager,
        const armnn::ITensorHandleFactory& tensorHandleFactory,
        float beta);
  4. Implement the unit tests and add template specializations for each data type your layer supports in the source file. The following code shows an example implementation of a unit test for the SoftmaxLayer using a Float32 data type. If supported by the new layer, you must add tests for other data types like Float16, Signed32, QAsymmS8, QAsymm16, and QAsymmU8:
    template<armnn::DataType ArmnnType, typename T = armnn::ResolveType<ArmnnType>>
    LayerTestResult<T, 2> SimpleSoftmaxTestImpl(
        armnn::IWorkloadFactory& workloadFactory,
        const armnn::IBackendInternal::IMemoryManagerSharedPtr& memoryManager,
        const armnn::ITensorHandleFactory& tensorHandleFactory,
        float beta)
    {
        using std::exp;
        const armnn::TensorShape inputShape{ 2, 4 };
    
        float x0[4] = { exp((0.f - 1.0f) * beta), exp((1.0f - 1.0f) * beta),
                        exp((0.0f - 1.0f) * beta), exp((0.0f - 1.0f) * beta) };
        float sum0 = x0[0] + x0[1] + x0[2] + x0[3];
        float x1[4] = { exp((0.5f - 0.5f) * beta), exp((0.0f - 0.5f) * beta),
                        exp((0.0f - 0.5f) * beta), exp((0.0f - 0.5f) * beta) };
        float sum1 = x1[0] + x1[1] + x1[2] + x1[3];
    
        const std::vector<float> outputData = { x0[0] / sum0, x0[1] / sum0, x0[2] / sum0, x0[3] / sum0,
                                                x1[0] / sum1, x1[1] / sum1, x1[2] / sum1, x1[3] / sum1 };
    
        const std::vector<float> inputData =
                {
                    0.f, 1.f, 0.f, 0.f,
                    .5f, 0.f, 0.f, 0.f,
                };
    
        return SimpleSoftmaxBaseTestImpl<ArmnnType, 2>(workloadFactory, memoryManager, tensorHandleFactory, beta,
                                                       inputShape, outputData, inputData);
    }
    
    ...
    
    LayerTestResult<float,2> SimpleSoftmaxTest(
        armnn::IWorkloadFactory& workloadFactory,
        const armnn::IBackendInternal::IMemoryManagerSharedPtr& memoryManager,
        const armnn::ITensorHandleFactory& tensorHandleFactory,
        float beta)
    {
        return SimpleSoftmaxTestImpl<armnn::DataType::Float32>(workloadFactory, memoryManager, tensorHandleFactory, beta);
    }
  5. Include the test suite in backends/backendsCommon/test/LayerTests.hpp. The following code shows this inclusion for the example SoftmaxLayer:
    #include <backendsCommon/test/layerTests/SliceTestImpl.hpp>
    +#include <backendsCommon/test/layerTests/SoftmaxTestImpl.hpp>
    #include <backendsCommon/test/layerTests/SpaceToBatchNdTestImpl.hpp>
  6. Add the unit tests to backends/<BackendName>/test/<BackendName>LayerTests.cpp. <BackendName> is the backend that you are targeting for your layer.
Previous Next