Install and operate

Multi-Cloud HyperShift Architecture

Sentinel's HyperShift infrastructure supports deploying and testing on multiple cloud providers: AWS and Google Cloud Platform (GCP).

Overview#

HyperShift creates hosted OpenShift clusters on managed cloud infrastructure. Sentinel uses HyperShift to: - Run E2E tests on real OpenShift clusters - Test Sentinel components across clouds - Provide infrastructure-as-code for cluster provisioning - Manage cluster lifecycle (creation, testing, cleanup)

Supported Clouds#

Cloud	Status	Management Cluster	Hosted Cluster	Docs
AWS	✅ Production	AWS	AWS	AWS.md
GCP	Planned, not implemented (unverified draft)	GKE (separate)	GCP	GCP.md

Management Cluster: Hosts HyperShift control plane. AWS hosted clusters use the AWS management cluster; GCP (not yet implemented) needs its own GKE-based management cluster.
Hosted Cluster: The OpenShift cluster being tested

Cloud Selection#

How It Works#

The system automatically detects which cloud to use based on which secrets are set:

if [ -n "$AWS_ACCESS_KEY_ID" ]; then
    # Use AWS
    .github/scripts/hypershift/cloud/aws/create-cluster.sh
elif [ -n "$GCP_PROJECT_ID" ]; then
    # Use GCP
    .github/scripts/hypershift/cloud/gcp/create-cluster.sh
fi

No configuration needed — the scripts detect your cloud automatically.

Local Development#

Set your cloud choice via environment file:

AWS (default):

source .env.sentinel-hypershift-aws
# Defines: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, etc.

GCP:

source .env.sentinel-hypershift-gcp
# Defines: GCP_PROJECT_ID, GCP_SERVICE_ACCOUNT_JSON, etc.

Then use the same commands:

.github/scripts/hypershift/create-cluster.sh
# Automatically detects AWS or GCP and creates cluster

CI/CD#

GitHub Actions uses matrix strategy to test both clouds in parallel:

strategy:
  matrix:
    cloud: [aws, gcp]

Both clouds run simultaneously, each with their own secrets and cluster naming.

Key Concepts#

Cluster Naming#

Clusters are named with the pattern:

${MANAGED_BY_TAG}-${CLOUD}-${SUFFIX}

Examples: - AWS: sentinel-hypershift-custom-aws-pr529 - GCP: sentinel-hypershift-custom-gcp-pr529

Purpose: Namespace clusters by cloud and prevent naming conflicts.

IAM Scoping & Cost Tracking#

Both clouds use resource tags to: - Limit credentials to specific resources (IAM scoping) - Track costs per project/team - Cleanup resources matching a tag

Tag format:

sentinel.io/managed-by: sentinel-hypershift-custom

Management Cluster#

Each cloud needs its own management cluster. Upstream documents a GKE Autopilot management cluster for GCP; it cannot be hosted from the AWS management cluster.

Directory Structure#

.github/scripts/hypershift/
├── shared/lib/                     # Cloud-agnostic utilities
│   ├── logging.sh                 # Color output, log functions
│   ├── validation.sh              # Common validations
│   ├── cluster-naming.sh           # Cluster naming logic
│   └── kubeconfig.sh              # K8s config handling
│
├── cloud/aws/                      # AWS implementation
│   ├── create-cluster.sh           # Create AWS cluster
│   ├── destroy-cluster.sh          # Destroy AWS cluster
│   ├── setup-credentials.sh        # AWS credential setup
│   ├── preflight-check.sh          # AWS quota validation
│   ├── policies/                   # AWS IAM policies
│   └── terraform/                  # AWS Terraform configs
│
├── cloud/gcp/                      # GCP implementation (Phase 3+)
│   ├── create-cluster.sh           # Create GCP cluster
│   ├── destroy-cluster.sh          # Destroy GCP cluster
│   ├── setup-credentials.sh        # GCP credential setup
│   ├── preflight-check.sh          # GCP quota validation
│   └── iam-roles/                  # GCP IAM roles
│
├── lib/                            # Dispatchers & routing
│   ├── dispatcher.sh               # Routes to cloud-specific scripts
│   ├── cloud-selector.sh           # Detects cloud provider
│   └── config-loader.sh            # Loads cloud config
│
├── ci/                             # CI orchestration
│   ├── 00-validate-secrets.sh      # Validate credentials
│   ├── 01-setup-credentials.sh     # Setup cloud credentials
│   ├── 02-create-cluster.sh        # Create cluster (dispatcher)
│   ├── 03-run-tests.sh             # Run E2E tests
│   ├── 04-destroy-cluster.sh       # Destroy cluster (dispatcher)
│   └── shared/                     # CI utilities
│
└── create-cluster.sh               # Wrapper (backward compatibility)

Command Flow#

When you run .github/scripts/hypershift/create-cluster.sh:

create-cluster.sh (wrapper)
    ↓ detect cloud
lib/dispatcher.sh
    ↓ route based on cloud
cloud/aws/create-cluster.sh  OR  cloud/gcp/create-cluster.sh
    ↓ execute cloud-specific logic
AWS HyperShift OR GCP HyperShift
    ↓ deploy
OpenShift cluster created ✅

Secrets Management#

Required Secrets#

Shared (both clouds):

HYPERSHIFT_MGMT_KUBECONFIG     # Management cluster kubeconfig
PULL_SECRET                     # OpenShift pull secret (Red Hat)
BASE_DOMAIN                     # DNS domain (e.g., sentinel-dev.example.com)
MANAGED_BY_TAG                 # Resource scoping tag

AWS-Specific:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_REGION                      # e.g., us-east-1
HCP_ROLE_NAME                  # IAM role for hosted control planes

GCP-Specific:

GCP_PROJECT_ID
GCP_REGION                      # e.g., us-central1
GCP_SERVICE_ACCOUNT_JSON       # Base64 encoded service account key
GCP_HCP_ROLE_NAME              # IAM role for hosted control planes

How to Provide Secrets#

Local Development (.env file):

# .env.aws
CLOUD_PLATFORM=aws
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
# ... other secrets

CI/CD (GitHub Actions repository settings): - Add secrets to Settings → Secrets and variables → Actions - GitHub automatically injects them as environment variables - One set per cloud (AWS and GCP secrets don't conflict)

Validating Secrets#

# Validate before running anything
.github/scripts/hypershift/ci/00-validate-secrets.sh

# Output: "All secrets validated for aws" ✅
# Or: "All secrets validated for gcp" ✅
# Or: "ERROR: Cannot detect cloud provider" ❌

Workflow: Create a Cluster#

Step 1: Set Up Credentials#

# Load your cloud's credentials
source .env.sentinel-hypershift-aws
# Or
source .env.sentinel-hypershift-gcp

# Validate they're correct
.github/scripts/hypershift/ci/00-validate-secrets.sh

Step 2: Create Cluster#

# Cloud is auto-detected, same command for both
.github/scripts/hypershift/ci/02-create-cluster.sh

# Or specify a cluster suffix
.github/scripts/hypershift/ci/02-create-cluster.sh pr529

# Cluster name: sentinel-hypershift-custom-aws-pr529 (AWS)
# Or:           sentinel-hypershift-custom-gcp-pr529 (GCP)

Takes 15-20 minutes - HyperShift creates infrastructure and bootstraps OpenShift.

Step 3: Access Cluster#

# Kubeconfig is automatically saved
export KUBECONFIG=~/.clusters/hcp/sentinel-hypershift-custom-aws-pr529/auth/kubeconfig

# Verify cluster is ready
kubectl get nodes

Step 4: Run Tests#

# E2E tests automatically run in CI
# For local testing:
pytest sentinel/tests/e2e/ -v

Step 5: Destroy Cluster#

# Same command for both clouds (auto-detected)
.github/scripts/hypershift/ci/04-destroy-cluster.sh pr529

# Cleans up all AWS or GCP resources
# Removes cluster kubeconfig

Comparing AWS and GCP#

Infrastructure#

Aspect	AWS	GCP
Instances	EC2	Compute Engine
DNS	Route53	Cloud DNS
Load Balancer	ELB/ALB	Cloud Load Balancer
VPC/Networking	VPC	VPC
IAM	AWS IAM	Google IAM
Service Account	IAM User	Service Account

Common Operations#

Operation	AWS	GCP
View instances	`aws ec2 describe-instances`	`gcloud compute instances list`
View DNS records	`aws route53 list-resource-record-sets`	`gcloud dns record-sets list`
View quotas	AWS Console → Service Quotas	GCP Console → Quotas
Check costs	AWS Console → Billing	GCP Console → Billing

Cluster Creation Time#

AWS: 15-20 minutes
GCP: 15-20 minutes

Times are similar; GCP may be slightly faster depending on quota and region.

Troubleshooting#

Cloud Detection Issues#

Problem: "ERROR: Cannot detect cloud provider"

Solution: Verify one of these is set:

echo $AWS_ACCESS_KEY_ID    # AWS
echo $GCP_PROJECT_ID       # GCP

If both or neither are set, explicitly set CLOUD_PLATFORM:

export CLOUD_PLATFORM=aws
# or
export CLOUD_PLATFORM=gcp

Quota Exceeded#

AWS: EC2 instance limit, VPC limit, Elastic IP limit
GCP: Compute Engine quota, Load Balancer quota

Solution:

# Check AWS quotas
aws service-quotas list-service-quotas --service-code ec2

# Check GCP quotas
gcloud compute project-info describe --project=$GCP_PROJECT_ID | grep QUOTA

Cluster Creation Fails#

Common causes: 1. Credentials expired or invalid 2. Insufficient quotas 3. Network issue (can't reach management cluster) 4. DNS domain not delegated correctly

Debug:

# Check credentials
aws sts get-caller-identity          # AWS
gcloud auth list                     # GCP

# Check kubeconfig to management cluster
kubectl cluster-info                 # Requires KUBECONFIG set

# Check cluster logs in CI
# Look at GitHub Actions workflow logs

Cleanup Issues#

Problem: Resources not deleted (zombie EC2/GCP instances)