Data workflows

The HERE platform organizes data according to three key concepts: catalog, layer, and partition. The platform hosts a number of catalogs, each with one or more layers, which are divided into partitions.

For more information on these concepts, see the Data API.

The OLP CLI provides tools for managing data, such as:

  • List all catalogs to which you have access
  • Create a new catalog and layer
  • Upload data to a catalog

For more details, see catalog commands.

List all catalogs

To list the catalogs to which you have access, enter the olp catalog list command.

olp catalog list

The command lists the HRNs of the catalogs to which you have access, each in a new line.

Create a new catalog and layer

Follow the steps below to create a new catalog and layer.

  1. Enter the olp catalog create command.

    Linux
    Windows
    olp catalog create first-catalog-example-id first-catalog-example \
        --summary "This catalog contains a sample layer containing data in CSV format"
    olp catalog create first-catalog-example-id first-catalog-example ^
        --summary "This catalog contains a sample layer containing data in CSV format"
  2. To add a layer to your catalog, enter the olp catalog layer add command.

    Linux
    Windows
    olp catalog layer add hrn:here:data:::first-catalog-example-id example-layer-id \
         "Example layer" --versioned --summary "This layer contains simple data in CSV format" \
         --description "A longer description of the layer contents" --content-type=text/csv
    olp catalog layer add hrn:here:data:::first-catalog-example-id example-layer-id ^
        "Example layer" --versioned --summary "This layer contains simple data in CSV format" ^
        --description "A longer description of the layer contents" --content-type=text/csv

    The command above adds a versioned layer with the ID example-layer-id and name Example layer to the hrn:here:data:::first-catalog-example-id catalog. The content type of this layer is set to text/csv.

    For Example layer, the following parameters were not explicitly used and set to their default values:

    • --content-encoding uncompressed The content encoding of the layer. If this parameter is specified, all data in a layer is compressed by the specified algorithm. The default value is uncompressed, which means no compression.
    • --partitioning generic The partitioning scheme of the layer, either heretile or generic. For heretile, you can specify tile levels separated by a comma, such as heretile:level1,level2. The platform supports tile levels from 1 to 15. For more information on HERE Tile Partitioning, see the Calculate Partition Tile IDs tutorial. The default value is generic.
  3. To verify the catalog configuration, enter the olp catalog show command.

    
        olp catalog show hrn:here:data:::first-catalog-example-id --json
        

    The command displays the following results.

    
        {
            "summary": "This catalog contains sample layers containing simple text",
            "owner": {
                "creator": {"id": "mzLcb1rL8nskvDQpCFEF"},
                "organisation": {"id": "olp-here"}
            },
            "billingTags": [],
            "hrn": "hrn:here:data:::first-catalog-example-id",
            "created": "2018-03-26T12:48:10.315Z",
            "name": "first-catalog-example",
            "layers": [{
                "summary": "This layer contains simple data in CSV format",
                "volume": {
                     "volumeType": "durable"
                },
                "layerType": "versioned",
                "billingTags": [],
                "name": "Example layer",
                "description": "A longer description of the layer contents",
                "partitioningScheme": "generic",
                "partitioning": {
                     "scheme": "generic"
                },
                "id": "example-layer-id",
                "contentType": "text/csv",
                "tags": []
            }],
            "description": "",
            "notifications": {
                "enabled": false
             },
            "marketplaceReady": false,
            "id": "first-catalog-example-id",
            "version": -1,
            "tags": []
        }
        

    From the command output above, you can see that a catalog with HRN hrn:here:data:::first-catalog-example-id has been successfully created. Apparently, it contains one layer of versioned type, so it stores slowly changing data that must remain logically consistent with other layers in the catalog.

    This layer has the ID example-layer-id and the following properties:

    • volumeType: durable Describes storage details of the layer. This layer has durable storage volume type because in versioned layers data is unchanged in the scope of one version.
    • billingTags: [] List of billing tags - a free-form tag that is used for grouping billing records together. For this layer, billing tags are not present.
    • partitioningScheme: generic This parameter defines the structure and encoding of the data by associating a schema with the layer. For this layer, the partitioning scheme is set to generic, which is the simplest form of partitioning. In this case, partition names have no semantic meaning. For more information, see the Partitions documentation.
    • contentType: text/csv The MIME type of the data that is stored in this layer. MIME type text/csv has been created for the handling of text-based fields.

    For more information on layers, see the Data User Guide.

Upload data to a catalog

  1. Create a file with the details you want to upload, such as partition-0.csv in the following example.

     City,Country
     Berlin,Germany
     Seattle,USA
     Chicago,USA
     Boston,USA
    
  2. Enter the olp catalog layer partition put command.

    Linux
    Windows
    olp catalog layer partition put hrn:here:data:::first-catalog-example-id \
         example-layer-id --partitions testPart0:<path_to_file>/partition-0.csv
    olp catalog layer partition put hrn:here:data:::first-catalog-example-id ^
        example-layer-id --partitions testPart0:<path_to_file>\partition-0.csv

    The following lines appear.

     Uploading the files into partitions.
    
      100% [=======================================] 1/1 (0:00:01 / 0:00:00)
     The data was successfully added to the partitions: [testPart0]
    
  3. To verify the partition, enter the olp catalog layer partition list command.

    
        olp catalog layer partition list hrn:here:data:::first-catalog-example-id example-layer-id
        

    The console displays the following text.

     Available partitions:
    
     partition           size                checksum
     testPart0           23                  665d2817c51cbccd4312eb22699163c05b9d5a0a
    
     Total size: 23 B
    
  4. To retrieve the partition-0.csv file from the olpcli-example-catalog-id, enter the olp catalog layer partition get command.

    Linux
    Windows
    olp catalog layer partition get \
         hrn:here:data:::first-catalog-example-id example-layer-id --partitions testPart0
    olp catalog layer partition get ^
        hrn:here:data:::first-catalog-example-id example-layer-id --partitions testPart0

    The console should contain the text inside the partition-0.csv file:

     City,Country
     Berlin,Germany
     Seattle,USA
     Chicago,USA
     Boston,USA
    

To get a full list of the available commands, enter olp --help.

results matching ""

    No results matching ""