1. Install self-hosted EraSearch

Estimated time to read: 5 minutes

Self-hosted EraSearch lets you run and manage EraSearch on your own cloud infrastructure. Use it to integrate EraSearch into your existing setup, lower costs, and get the most out of your log data.

This two-part series is intended for developer audiences, and it outlines how to install and use self-hosted EraSearch. In this series, you’ll:

• Set up and run EraSearch on your cloud infrastructure.
• Write data to EraSearch.
• Explore and query data using the EraSearch API.

Before you begin¶

Before you start, reach out to us at Era Software to get the following:

• eradb-x.x.x.tgz - Compressed files with Helm charts for EraSearch.
• values-eradb.yaml - The Helm chart values file for EraSearch, customized for your deployment.
• eradb-registry-xxxx.json - Authentication credentials for Era Software's container registry.

This guide assumes you're comfortable with Kubernetes, Helm, and EraSearch's high-level architecture. It also assumes you have the following in your environment:

• Kubernetes version 1.20+ with persistent volumes, and Helm.

• kubectl and the Helm CLI.

  Make sure they're installed on the machine you'll be deploying from.

• An object storage provider.

  Self-hosted EraSearch works with Amazon Web Services (AWS) S3, Google Cloud Storage, Microsoft Azure Blob Storage, or an API-compatible alternative.

Step 1: Set up the Kubernetes namespace¶

Follow these steps to create a Kubernetes namespace for the EraSearch deployment:

1. In your terminal, enter this command to create a new namespace, replacing NAMESPACE_NAME with your namespace name.

   $ kubectl create namespace NAMESPACE_NAME
   

2. Create a secret for storing Era Software's container registry credentials in the namespace. Enter this command, replacing:

   • NAMESPACE_NAME with the namespace name you specified above.
   • REGISTRY_SECRET with the name of the JSON credential file you got from Era Software.
   • REGISTRY_EMAIL with the email address in that same file.

   $ kubectl create secret docker-registry eradb-registry \
       -n NAMESPACE_NAME \
       --docker-server=https://us.gcr.io \
       --docker-username=_json_key \
       --docker-password="$(cat path/to/REGISTRY_SECRET)" \
       --docker-email=REGISTRY_EMAIL
   

Step 2: Configure the Helm deployment¶

Make sure the values in values-eradb.yaml are right for your EraSearch deployment.

For every database service, review these values:

• image.repository

  You may need to update the values if you're mirroring images from an internal registry instead of directly pulling them from Era Software's registry.

• replicaCount, resources.cpu, and resources.memory

  Make sure the values are valid for your Kubernetes cluster and deployment size.

Next, review these maxwell.treasurer values, making sure they're valid for your environment and object storage budget:

• monthly_budget

  The maximum USD amount you want to spend on object storage per month.

• batch_size

  The maximum batch size, in bytes, after which EraSearch flushes data to object storage.

• batch_delay_ms

  The time, in milliseconds, after which EraSearch flushes data to object storage.

For AWS deployments¶

If you're using AWS, also review the following:

• maxwell.s3.region and maxwell.s3.bucket

  Set the values to the right bucket and region for your deployment.

• maxwell.extraEnv

  Make sure this section includes valid AWS credentials under the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. For production deployments, Era Software recommends using Kubernetes service accounts with IAM pod roles instead of environment-provided credentials.

• quarry.persistence.storageClass

  Set the value to the supported storage class. Era Software recommends starting with gp2. Visit AWS's Storage classes for more information.

For Microsoft Azure deployments¶

If you're using Microsoft Azure, also review the following:

• maxwell.s3

  Make sure the section specifies your Azure storage bucket, region, and endpoint (including the URI and SAS token). You may also want to check the network security settings for your storage account, ensuring EraSearch deployment requests can authenticate.

  s3:
   bucket: your-container-name
   region: eastus2
   endpoint: https://XXXX.blob.core.windows.net/?sv=2020-08-04&ss=b&srt=sco&sp=rwdlacitfx&se=2023-03-04T14:33:22Z&st=2022-03-04T06:33:22Z&spr=https&sig=XXXXXXXX
  

  Note

  While the name s3 refers to Amazon's object storage service, the specification still works with Microsoft Azure Blob Storage.

• maxwell.extraEnv

  In this section, set the following array elements:

  extraEnv:
   - name: MAXWELL_STORAGE
     value: azure_blob
   - name: MAXWELL_AZURE_CONTAINER
     value: your-container-name
  

• quarry.persistence.storageClass

  Set the value to supported storage class. Era Software recommends starting with managed-premium unless you need provisioned IOPS. Visit Microsoft Azure's Storage classes for more information.

Step 3: Install EraSearch¶

With the values set and validated, follow these steps to install EraSearch:

1. Enter the command below in your terminal, replacing:

   • NAME with a name for the EraSearch database release (for example, era).
   • X.X.X with the version of the Helm chart you got from Era Software.
   • NAMESPACE_NAME with the Kubernetes namespace you created above.

   $ helm install NAME ./eradb-X.X.X.tgz \
       --values ./values-eradb.yaml \ 
       --namespace NAMESPACE_NAME
   

2. The installation takes a few minutes to complete. Enter this command to check its status:

   $ kubectl rollout status deployment NAME-quarry-deployment -n NAMESPACE_NAME
   

   Once the installation completes, you'll see this message:

   deployment "NAME-quarry-deployment" successfully rolled out
   

Step 4: Verify your installation¶

Follow these steps to verify your EraSearch installation:

1. Connect to the quarry-deployment service. Enter the command below to open a local port forward, replacing:

   • NAMESPACE_NAME with the Kubernetes namespace you used above.
   • NAME with the name of the EraSearch database release you used above.

   $ kubectl port-forward -n NAMESPACE_NAME svc/NAME-quarry-deployment 9200:9200
   

2. Visit http://localhost:9200. When EraSearch is deployed and the API is ready for data, you'll see this JSON document with metadata about your instance:

   {
      "name": "XXXXX",
      "cluster_name": "erasearch_cluster",
      "cluster_uuid": "XXXXX",
      "version": {
         "number": "XXXXX",
         [...]
         "lucene_version": "XXXXX"
      },
      "tagline": "You Know, for Search"
   }
   

You're all set. You've now installed self-hosted EraSearch in your environment.

Next steps¶

Next, visit 2. Write and query data to get the basics and use the EraSearch REST API. If you're ready to integrate EraSearch into your existing setup, visit the list of tools for writing real-time data.