Pipeline Quotas & Limits

When configuring a pipeline template, you need to choose the amount of resources to allocate to a Supervisor and Worker. Resources are allocated as units, each containing a pre-defined number of CPUs, RAM size, and disk space.

For each Supervisor or Worker (also referred to as supervisor units and worker units), the following size limits apply:

  • Minimum Size: 1 unit
  • Maximum Size: 15 units

A pipeline can contain:

  • Number of Supervisors: 1
  • Number of Workers: 199

The amount of resources per unit is defined by the Resource profile that you can select for each pipeline template.

Resource profiles

The HERE platform supports several resource profiles with different combinations of CPU number, RAM size, and disk space per unit to ensure the most effective resource consumption. There are memory-intensive, CPU-intensive, and default resource profiles. Depending on the data processing type and the tasks that the pipeline is designed to perform, you can choose a resource profile that is the most optimal for your project goals.

The following default resource profiles are available:

  • Stream pipelines: 1 CPU, 7 GB RAM, and 8 GB disk space per unit
  • Batch pipelines: 1 CPU, 7 GB RAM, and 32 GB disk space per unit

Memory-intensive and CPU-intensive resource profiles contain a higher or lower RAM amount, respectively. To find out which resource profiles are currently available on the HERE platform and the corresponding resource-per-unit ratios, make a request to one of the following endpoints:




The response will contain a list of the available resource profiles with the amount of resources per unit:

    "availableResourceProfiles": [
            "id": "profileId",
            "label": "User-friendly label for display",
            "name": "Description of the profile",
            "cpuPerUnit": 1,
            "memoryInGiBPerUnit": 2,
            "diskInGiBPerUnit": 3

To find the resource profiles available for a particular runtime environment, query the following endpoints:




The response contains a list of runtime environments showing the available and default resource profiles:

    "availableRuntimes": [
            "id": "batch-x.y.z",
            "availableResourceProfileIds": [
            "defaultWorkerResourceProfileId": "HS1",
            "defaultSupervisorResourceProfileId": "HS1",
            "maxWorkerUnits": 15,
            "maxSupervisorUnits": 15,
            "description": "Spark x.y.z, Java x, Scala y",
            "deprecationPeriodAnnounced": "2007-12-03T10:15:30+01:00",
            "deprecationPeriodEnd": "2007-12-03T10:15:30+01:00",
            "displayName": "Batch x.y.z"

Caution: Limited Resource Requirements

HERE platform limits are still applicable. A pipeline can only consume a maximum of 200 CPU and 1.4 TB RAM. In other words:
Size of Supervisor + (Size of Each Worker * Number of Workers) ≤ 200 CPU AND 1.4 TB RAM
If you experience any resource provisioning issues within these limits, you should contact HERE Support/Services.

For example:

For a Spark batch pipeline, choose the following units for resource allocation of the Supervisor (Spark Driver) and Worker (Spark Executor):

  • Spark Driver: 2 units (equal to 2 CPU, 14 GB RAM, 64 GB Disk Space)
  • Spark Executors:
    • Each Executor size: 3 units (equal to 3 CPU, 21 GB RAM, 96 GB Disk Space)
    • Number of Executors: 2
  • Total Resource Requirement: 8 units (equal to 8 CPU, 56 GB RAM, 256 GB Disk Space)

results matching ""

    No results matching ""