Install and operate

AWS HyperShift Setup & Operation

Complete guide for setting up and managing Sentinel HyperShift clusters on AWS.

Prerequisites#

Tools Required#

Tool	Version	Purpose
AWS CLI	2.x	AWS resource management
kubectl	1.28+	Kubernetes cluster interaction
jq	Latest	JSON processing
Bash	3.2+	Script execution
Python	3.11+	E2E tests

macOS Installation#

brew install awscli kubectl jq python@3.11

Linux (Ubuntu/Debian)#

sudo apt-get update
sudo apt-get install -y awscli kubectl jq python3.11

# Or use Snap for AWS CLI
sudo snap install aws-cli --classic

AWS Account Requirements#

Admin access (for one-time setup only)
EC2 quota: 10+ instances
VPC quota: 5+ VPCs
Elastic IP quota: 10+ addresses
Load Balancer quota: 10+ load balancers

Check your quotas:

aws service-quotas list-service-quotas --service-code ec2 \
  --query 'ServiceQuotas[?QuotaName==`Running On-Demand EC2 instances`]'

One-Time Setup (AWS Admin)#

These steps run once to prepare your AWS account. You need admin access only for this phase.

Step 1: Set Admin Credentials#

# Get your AWS root or admin credentials from AWS console
export AWS_ACCESS_KEY_ID="<your-admin-access-key>"
export AWS_SECRET_ACCESS_KEY="<your-admin-secret-key>"
export AWS_REGION="us-east-1"  # Or your preferred region

Verify:

aws sts get-caller-identity
# Output shows your account info if credentials work

You need cluster-admin access to the existing HyperShift management cluster:

# Get management cluster kubeconfig (from whoever set it up)
export KUBECONFIG=~/.kube/hypershift_mgmt

# Or download from AWS Console → EKS → your-cluster

# Verify access
kubectl cluster-info
kubectl get ns | grep hypershift

Step 3: Create Scoped Credentials#

This creates an IAM user with limited permissions (good practice):

# Run the credential setup script
./.github/scripts/hypershift/cloud/aws/setup-credentials.sh

# This:
# 1. Creates IAM user (sentinel-hypershift-ci-user)
# 2. Creates IAM roles for hosted control planes
# 3. Creates OpenShift service account
# 4. Generates access keys
# 5. Outputs: .env.sentinel-hypershift-custom

Output: .env.sentinel-hypershift-custom (git-ignored)

# Contents:
export CLOUD_PLATFORM=aws
export AWS_ACCESS_KEY_ID=AKIA...
export AWS_SECRET_ACCESS_KEY=...
export AWS_REGION=us-east-1
export HYPERSHIFT_MGMT_KUBECONFIG=~/.kube/hypershift_mgmt
export PULL_SECRET=$(cat ~/.docker/config.json)
export BASE_DOMAIN=sentinel-dev.example.com
export MANAGED_BY_TAG=sentinel-hypershift-custom
export HCP_ROLE_NAME=sentinel-hypershift-custom-hcp-role

Step 4: Verify Setup#

# Load the new scoped credentials
source .env.sentinel-hypershift-custom

# Run preflight check
./.github/scripts/hypershift/cloud/aws/preflight-check.sh

# Output should show:
# ✓ AWS credentials valid
# ✓ Management cluster accessible
# ✓ Quotas sufficient
# ✓ Pull secret present
# ✓ Base domain configured

✅ One-time setup complete! You now have scoped credentials for daily development.

Daily Usage (Scoped Credentials)#

After one-time setup, use these scoped credentials for all cluster operations.

Quick Start#

# 1. Load your credentials
source .env.sentinel-hypershift-custom

# 2. Create a cluster
./.github/scripts/hypershift/create-cluster.sh

# Wait 15-20 minutes...
# ✓ Cluster created
# ✓ Kubeconfig saved to ~/.clusters/hcp/<cluster-name>/

# 3. Use the cluster
export KUBECONFIG=~/.clusters/hcp/sentinel-hypershift-custom-aws-<suffix>/auth/kubeconfig
kubectl get nodes

# 4. Run tests
uv run pytest sentinel/tests/e2e/ -v

# 5. Cleanup
./.github/scripts/hypershift/destroy-cluster.sh <suffix>

Cluster Naming#

Clusters are named: sentinel-hypershift-custom-aws-${SUFFIX}

Suffix options:

# Default: Your username (automatically sanitized)
./.github/scripts/hypershift/create-cluster.sh
# → sentinel-hypershift-custom-aws-ladas

# Custom suffix
./.github/scripts/hypershift/create-cluster.sh pr529
# → sentinel-hypershift-custom-aws-pr529

# Random suffix
CLUSTER_SUFFIX="" ./.github/scripts/hypershift/create-cluster.sh
# → sentinel-hypershift-custom-aws-abc123

Accessing the Cluster#

# List available kubeconfigs
ls ~/.clusters/hcp/

# Set kubeconfig for your cluster
export KUBECONFIG=~/.clusters/hcp/sentinel-hypershift-custom-aws-pr529/auth/kubeconfig

# Verify access
kubectl cluster-info
kubectl get nodes
kubectl get pods -A

Destroying Clusters#

# Destroy specific cluster
./.github/scripts/hypershift/destroy-cluster.sh pr529

# Or by full name
./.github/scripts/hypershift/destroy-cluster.sh sentinel-hypershift-custom-aws-pr529

Important: Always destroy test clusters to avoid AWS costs!

Configuration#

Cluster Parameters#

Control cluster characteristics via environment variables:

# Number of worker nodes (default: 2)
export REPLICAS=3
./.github/scripts/hypershift/create-cluster.sh

# Instance type (default: m5.xlarge)
export INSTANCE_TYPE=m5.2xlarge
./.github/scripts/hypershift/create-cluster.sh

# OpenShift version (default: 4.20.11)
export OCP_VERSION=4.19.12
./.github/scripts/hypershift/create-cluster.sh

# Autoscaling (default: min=2, max=5)
export AUTOSCALE_MIN=1
export AUTOSCALE_MAX=10
./.github/scripts/hypershift/create-cluster.sh

# Disable autoscaling
export AUTOSCALE_MIN=""
export AUTOSCALE_MAX=""
./.github/scripts/hypershift/create-cluster.sh

AWS Region#

# Create cluster in different region
export AWS_REGION=us-west-2
./.github/scripts/hypershift/create-cluster.sh

Supported regions: Any AWS region with sufficient quotas

CI/CD Integration#

GitHub Actions Matrix#

Both AWS and GCP run simultaneously:

strategy:
  matrix:
    cloud: [aws, gcp]
  max-parallel: 2

Each cloud gets its own secrets and creates clusters independently.

Secrets Configuration#

Go to Repository Settings → Secrets and variables → Actions
Add these secrets (same names as .env file): - AWS_ACCESS_KEY_ID - AWS_SECRET_ACCESS_KEY - AWS_REGION - HYPERSHIFT_MGMT_KUBECONFIG (base64) - PULL_SECRET (JSON) - BASE_DOMAIN - MANAGED_BY_TAG - HCP_ROLE_NAME
Create Environments for e2e-aws and e2e-gcp with separate secret sets

Trigger E2E Tests#

# Automatic: Push to main or open PR
# Manual: GitHub Actions → e2e-hypershift → Run workflow

# Or via gh CLI
gh workflow run e2e-hypershift.yaml -f cluster_name=pr529

Troubleshooting#

Cluster Creation Fails#

Error: "ERROR: Quota exceeded"

Solution:

# Check your quotas
aws service-quotas list-service-quotas --service-code ec2 \
  --query 'ServiceQuotas[?QuotaName==`Running On-Demand EC2 instances`]'

# Request quota increase in AWS Console
# Services → Service Quotas → EC2 → desired quota → Request quota increase

Error: "ERROR: Cannot connect to management cluster"

Solution:

# Verify kubeconfig is set
echo $KUBECONFIG

# Verify it points to valid file
ls -la $KUBECONFIG

# Verify you can connect
kubectl cluster-info

Cluster Won't Delete#

Error: "ERROR: Timeout waiting for cluster deletion"

Solution:

# Check if control plane namespace is stuck
oc get ns clusters-sentinel-hypershift-custom-aws-<suffix>

# Force delete if stuck
oc delete ns clusters-sentinel-hypershift-custom-aws-<suffix> --grace-period=0 --force

# Check for orphaned resources
./.github/scripts/hypershift/cloud/aws/find-orphaned-resources.sh

# Manually delete via AWS Console if necessary
aws ec2 describe-instances --filters "Name=tag:kubernetes.io/cluster/sentinel-hypershift-custom-aws-*"

Access Denied Errors#

Error: "AccessDenied: User is not authorized..."

Solution:

# Verify credentials
aws sts get-caller-identity
# Should show IAM user, not root

# Check IAM permissions
aws iam get-user-policy --user-name sentinel-hypershift-ci-user --policy-name sentinel-policy

# If missing, rerun setup-credentials.sh with admin access

DNS Issues#

Error: "Could not resolve .sentinel-dev.example.com"

Solution:

# Verify BASE_DOMAIN is correct
echo $BASE_DOMAIN

# Verify domain is delegated to Route53
aws route53 list-hosted-zones | grep sentinel-dev

# Check created DNS records
aws route53 list-resource-record-sets --hosted-zone-id <zone-id>

Cost Management#

Monitoring Costs#

# AWS Console → Billing → Cost Explorer
# Filter by tag: sentinel.io/managed-by = sentinel-hypershift-custom

# Or via AWS CLI
aws ce get-cost-and-usage \
  --time-period Start=2026-10-01,End=2026-10-31 \
  --filter file://cost-filter.json

Cost Breakdown#

Per cluster (typical): - EC2 instances: $30-50/month - ELB: $15/month - VPC/ENI: $10/month - Route53: <$1/month - Total: ~$55-75/month

Cost Optimization#

Destroy unused clusters (biggest cost driver)
Use smaller instances for non-critical tests (t3.large instead of m5.xlarge)
Reduce node count (1 node instead of 2 if possible)
Schedule tests to run only during business hours

Debugging#

Cluster Status#

# Check if cluster is ready
oc get hostedcluster -n clusters

# Check control plane pod status
oc get pods -n clusters-<cluster-name>

# View cluster events
oc describe hostedcluster -n clusters <cluster-name>

Logs#

# HyperShift operator logs
oc logs -n hypershift-system deployment/hypershift-operator -f

# Control plane pod logs
oc logs -n clusters-<cluster-name> <pod-name>

# SSH to worker node (via AWS Console)
# EC2 → Instances → select instance → Connect → Session Manager

AWS Resources#

# View all EC2 instances with sentinel tags
aws ec2 describe-instances \
  --filters "Name=tag-key,Values=kubernetes.io/cluster/sentinel-hypershift-custom-aws-*"

# View all Route53 records
aws route53 list-resource-record-sets --hosted-zone-id <zone-id>

# View all security groups
aws ec2 describe-security-groups \
  --filters "Name=tag-key,Values=kubernetes.io/cluster/sentinel-hypershift-custom-aws-*"

Advanced: Custom VPC Configuration#

By default, HyperShift creates a new VPC for each cluster. To use existing VPC:

# Edit AWS Terraform to use existing VPC
# File: .github/scripts/hypershift/cloud/aws/terraform/variables.tf

# Set VPC ID and subnet IDs
export VPC_ID="vpc-12345"
export SUBNET_IDS="subnet-11111,subnet-22222"

./.github/scripts/hypershift/create-cluster.sh

Frequently Asked Questions#

Q: How long does cluster creation take?
A: 15-20 minutes typically. Monitor in CI logs or management cluster.

Q: Can I access the cluster while tests run?
A: Yes, kubeconfig is created immediately. Tests run concurrently.

Q: What happens to cluster after E2E tests?
A: It's automatically destroyed to save costs (unless you set skip_destroy=true).

Q: Can I use a different AWS region?
A: Yes, set AWS_REGION before creation.

Q: How much does a test cluster cost?
A: ~$2-3 per day (if running continuously). Always destroy when done.

Q: Can I modify a running cluster?
A: Yes, you have full kubectl access. Changes persist during test run.

Q: What if credentials expire?
A: Request new access keys from whoever manages the IAM user.

ARCHITECTURE.md - Multi-cloud overview
GCP.md - Google Cloud Platform setup (Phase 3+)
Troubleshooting - Common issues for both clouds
Rotating HyperShift CI credentials - Secrets management details

Support#

HyperShift Docs: https://hypershift-docs.readthedocs.io/
AWS HyperShift: https://hypershift-docs.readthedocs.io/en/latest/how-to/aws/
OpenShift Docs: https://docs.openshift.com/

AWS HyperShift Setup & Operation

Prerequisites#

Tools Required#

macOS Installation#

Linux (Ubuntu/Debian)#

AWS Account Requirements#

One-Time Setup (AWS Admin)#

Step 1: Set Admin Credentials#

Step 2: Login to Management Cluster#

Step 3: Create Scoped Credentials#

Step 4: Verify Setup#

Daily Usage (Scoped Credentials)#

Quick Start#

Cluster Naming#

Accessing the Cluster#

Destroying Clusters#

Configuration#

Cluster Parameters#

AWS Region#

CI/CD Integration#

GitHub Actions Matrix#

Secrets Configuration#

Trigger E2E Tests#

Troubleshooting#

Cluster Creation Fails#

Cluster Won't Delete#

Access Denied Errors#

DNS Issues#

Cost Management#

Monitoring Costs#

Cost Breakdown#

Cost Optimization#

Debugging#

Cluster Status#

Logs#

AWS Resources#

Advanced: Custom VPC Configuration#

Frequently Asked Questions#

Related Documentation#

Support#