Measuring usage

Estimated time to read: 3 minutes

Starting with EraSearch v1.23.0, an optional Usage Service is included with EraSearch. This service tracks the volume of data ingested per index for billing and monitoring purposes. The Usage Service automatically writes data to EraSearch, letting you generate reports on byte-level volumes of data ingested over time.

This page outlines how to deploy the Usage Service and how to configure it with EraSearch RBAC.

Deploying the Usage Service¶

To deploy the Usage Service alongside an EraSearch deployment:

1. Add the following usage section to your Helm values file (values-eradb.yaml) quarry block:

   quarry:
     usage:
       # ⭐️ Enable the Usage Service deployment as part of the EraSearch rollout
       enabled: true
   
       # ⭐️ Optionally override the image settings
       image:
         repository: us.gcr.io/eradb-registry/usage
         tag: '1.0.2'
       imagePullSecrets:
         - name: eradb-registry
   

2. Upgrade your EraSearch deployment with the command below. Successful upgrade commands return several deployment details and Happy Helming!.

   $ helm upgrade ${RELEASE_NAME} ./charts/eradb-X.Y.Z.tgz -n ${NAMESPACE} --values values-eradb.yaml
   

   The Usage Service now tracks data ingestion, and you can query the data in the usage index. If you're using EraSearch RBAC, continue to the next section to finish setting up the new service.

Configuring RBAC for the Usage Service¶

If you're using EraSearch RBAC, then you need to add an API key with both read and write privileges on the usage index to the Usage Service configuration. The steps below show how to do that with a Kubernetes secret.

1. Create a role and API key for the usage service. If using the eractl CLI tool, create the role and API key with the commands below. The last command writes the API key to the local file usage-key.json

   $ export ERACTL_URL="http://api.erasearch.example.com"  # URL of the EraSearch API
   $ export ERACTL_API_KEY="XXXXX"                         # Admin key
   
   $ eractl rbac create-read-write -i usage -o usage-key.json
   

2. To create the Kubernetes secret, collect the API key value from the api_key field from the usage-key.json file with:

   $ USAGE_API_KEY="$(cat usage-key.json | jq .api_key | xargs)"
   

3. With the API key created, create the Kubernetes secret with the kubectl command:

   USAGE_API_KEY="era_XXXXXXXXXXXXXXXXXXXXX"
   NAMESPACE="my-erasearch-namespace"
   
   echo "apiVersion: v1
   kind: Secret
   type: Opaque
   metadata:
     name: erasearch-usage
   stringData:
     usage-api-key: ${USAGE_API_KEY}
   " | kubectl apply -f - -n ${NAMESPACE}
   

4. Add the relevant information to the Helm values file under the secretName and secretKey options so that the Usage Service can use it:

   quarry:
     usage:
       [...]
       secretName: "erasearch-usage"   # ⭐️ Make sure this matches the secret name you specified above
       secretKey: "usage-api-key"      # Set this to the attribute name you created in the secret above, not the raw API key value
   

5. Issue a helm upgrade command as referenced in the section above to deploy the Usage Service with the updated configuration.

6. For security purposes, remove the local file you created in step 1 (usage-key.json).

You're all set. The Usage Service is tracking data ingestion and writing the data to the usage index.

Generating a Usage Service report¶

To generate a per-day usage report, issue the following curl command against your EraSearch instance:

# Set the necessary environment variables
ERA_URL="https://api.erasearch.example.com"
ERA_API_KEY="XXXXXXX"

# Generate the report
curl -XPOST "${ERA_URL}/usage/_search" -H "Content-Type: application/json" -H "Authorization: Bearer ${ERA_API_KEY}" --data-binary '{"size":0,"query":{"bool":{"filter":[]}},"aggs":{"3":{"terms":{"field":"namespace","min_doc_count":1},"aggs":{"2":{"date_histogram":{"field":"_ts","min_doc_count":0,"format":"epoch_millis","fixed_interval":"1d"},"aggs":{"1":{"sum":{"field":"write_bytes"}}}}}}}}' > erasearch_usage_report.$(date +%s).json

Once the command has completed, a per-day usage report will be written to the local file erasearch_usage_report.${TIMESTAMP}.json (where ${TIMESTAMP} is the unix timestamp when the report was generated). Send the resulting file to Era Software support for review.