AWS Instance with Intel Processors for Red Hat OpenShift

Amazon EC2 provides a wide selection of instance types optimized to run different workload categories. These instance types comprise varying combinations of CPU, memory, storage, and networking capacity and give you the flexibility to choose the appropriate mix of resources for your applications. Each instance type includes one or more instance sizes, allowing you to scale your resources to the requirements of your target workload.

In this article we will look into the following:

  1. Select from Intel instances available on AWS.
  2. Install OpenShift on AWS.
  3. Add a c5.2xlarge instance with 2nd Generation Intel Xeon Scalable Processor (code name Cascade Lake) to the OpenShift cluster.
  4. Deploy an application such that the pods run on the newly added node.

AWS Instances with Intel Processors

Here we can see the AWS instances and the Intel Xeon Processors associated with them.

Benefits of Intel-based AWS EC2 instances
Source: https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/cloudtech-brief-aws.pdf

Install OpenShift on AWS

By default the IPI or the Installer Provisioned Installer creates 3 x Master nodes (m5.large) and 3 x Worker nodes (m5.xlarge) on each Availability Zone.

Note, if we want to use different instance types with the installer we can customize the instance type for the aws installation as explained on https://access.redhat.com/solutions/3803201.

In OpenShift Container Platform, you can also install a customized cluster on Amazon Web Services (AWS). To customize the installation, you modify parameters in the install-config.yaml file before you install the cluster.

For this article we will use the default IPI installer on AWS.

% ./openshift-install create cluster --dir=mydir
INFO Credentials loaded from the "default" profile in file "/Users/mshetty/.aws/credentials"
INFO Consuming Install Config from target directory
WARNING Following quotas ec2/L-0263D0A3 (us-west-2) are available but will be completely used pretty soon.
INFO Creating infrastructure resources...
INFO Waiting up to 20m0s for the Kubernetes API at https://api.mytest.ocp4-test-mshetty.com:6443...
INFO API v1.20.0+bafe72f up
INFO Waiting up to 30m0s for bootstrapping to complete...
INFO Destroying the bootstrap resources...
INFO Waiting up to 40m0s for the cluster at https://api.mytest.ocp4-test-mshetty.com:6443 to initialize...
INFO Waiting up to 10m0s for the openshift-console route to be created...
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/Users/mshetty/AWS/4.7/mydir/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.mytest.ocp4-test-mshetty.com
INFO Login to the console with user: "kubeadmin", and password: "sbGIs-shhAM-qPtS7-PrwTn"
INFO Time elapsed: 33m2s

After the deployment completes, you can log into the cluster and see the Master and Worker nodes are ready.

% oc get nodes
NAME                                         STATUS  ROLES    AGE    VERSION
ip-10-0-135-85.us-west-2.compute.internal    Ready   worker   4h8m   v1.20.0+bafe72f
ip-10-0-155-129.us-west-2.compute.internal   Ready   master   4h18m  v1.20.0+bafe72f
ip-10-0-164-249.us-west-2.compute.internal   Ready   worker   4h10m  v1.20.0+bafe72f
ip-10-0-189-65.us-west-2.compute.internal    Ready   master   4h17m  v1.20.0+bafe72f
ip-10-0-193-29.us-west-2.compute.internal    Ready   master   4h18m  v1.20.0+bafe72f
ip-10-0-214-229.us-west-2.compute.internal   Ready   worker   4h8m   v1.20.0+bafe72f

With the “oc describe node” command gives us detailed information about the node. As we see below it gives us information on all the Labels, Capacity, System Info, etc associated with the node.

% oc describe node ip-10-0-214-229.us-west-2.compute.internal
Name:                  ip-10-0-214-229.us-west-2.compute.internal
Roles:                 worker
Labels:                beta.kubernetes.io/arch=amd64
                       beta.kubernetes.io/instance-type=m5.large
                       beta.kubernetes.io/os=linux
                       failure-domain.beta.kubernetes.io/region=us-west-2
                       failure-domain.beta.kubernetes.io/zone=us-west-2c
                       kubernetes.io/arch=amd64
                       kubernetes.io/hostname=ip-10-0-214-229
                       kubernetes.io/os=linux
                       node-role.kubernetes.io/worker=
                       node.kubernetes.io/instance-type=m5.large
                       node.openshift.io/os_id=rhcos
                       topology.ebs.csi.aws.com/zone=us-west-2c
                       topology.kubernetes.io/region=us-west-2
                       topology.kubernetes.io/zone=us-west-2c
Annotations:           csi.volume.kubernetes.io/nodeid: {"ebs.csi.aws.com":"i-0e960db6db9add711"}
                       machine.openshift.io/machine: openshift-machine-api/mytest-qd52c-worker-us-west-2c-6j9dl
                       machineconfiguration.openshift.io/currentConfig: rendered-worker-640ffae9c463a45ced757500a238cfcf
                       machineconfiguration.openshift.io/desiredConfig: rendered-worker-640ffae9c463a45ced757500a238cfcf
                       machineconfiguration.openshift.io/reason:
                       machineconfiguration.openshift.io/state: Done
                      volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 21 Apr 2021 10:45:04 -0700
Taints:                <none>
Unschedulable:         false
Lease:
     HolderIdentity:  ip-10-0-214-229.us-west-2.compute.internal
     AcquireTime:         <unset>
     RenewTime:           Wed, 21 Apr 2021 14:50:07 -0700
Conditions:
Type                 Status  LastHeartbeatTime                    LastTransitionTime                    Reason                        Message
----                 ------  -----------------                    ------------------                    ------                        -------
  MemoryPressure   False   Wed, 21 Apr 2021 14:46:47 -0700   Wed, 21 Apr 2021 10:45:03 -0700   
KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Wed, 21 Apr 2021 14:46:47 -0700   Wed, 21 Apr 2021 10:45:03 -0700   
KubeletHasNoDiskPressure    kubelet has no disk pressure
  PIDPressure          False   Wed, 21 Apr 2021 14:46:47 -0700   Wed, 21 Apr 2021 10:45:03 -0700   
KubeletHasSufficientPID     kubelet has sufficient PID available
  Ready                True    Wed, 21 Apr 2021 14:46:47 -0700   Wed, 21 Apr 2021 10:46:24 -0700   
KubeletReady                kubelet is posting ready status
Addresses:
  InternalIP:   10.0.214.229
  Hostname:    ip-10-0-214-229.us-west-2.compute.internal
  InternalDNS:  ip-10-0-214-229.us-west-2.compute.internal
Capacity:
  attachable-volumes-aws-ebs:  25
  cpu:                        2
  ephemeral-storage:          125293548Ki
  hugepages-1Gi:              0
  hugepages-2Mi:              0
  memory:                     7936744Ki
  pods:                       250
Allocatable:
  attachable-volumes-aws-ebs:  25
  cpu:                        1500m
  ephemeral-storage:          114396791822
  hugepages-1Gi:              0
  hugepages-2Mi:              0
  memory:                     6785768Ki
  pods:                       250
System Info:
  Machine ID:                                ec240bd55efb903fd58499d67a7fc761
  System UUID:                               ec240bd5-5efb-903f-d584-99d67a7fc761
  Boot ID:                                   6897e7a9-660e-43e5-b0a7-7b8a8b1bf9ce
  Kernel Version:                            4.18.0-240.15.1.el8_3.x86_64
  OS Image:                                  Red Hat Enterprise Linux CoreOS 47.83.202104010243-0 
(Ootpa)
  Operating System:                          linux
  Architecture:                              amd64
  Container Runtime Version:                 cri-o://1.20.2-4.rhaos4.7.gitd5a999a.el8
  Kubelet Version:                           v1.20.0+bafe72f
  Kube-Proxy Version:                        v1.20.0+bafe72f
ProviderID:                                  aws:///us-west-2c/i-0e960db6db9add711
Non-terminated Pods:                         (16 in total)
  Namespace                                  Name                                  CPU Requests  
CPU Limits  Memory Requests  Memory Limits  AGE
---------                                  ----                                  ------------  ----------  ---------------  -------------  ---
openshift-cluster-csi-drivers              aws-ebs-csi-driver-node-vvxwp         30m (2%)    0 (0%)         150Mi (2%)     0 (0%)         4h5m
openshift-cluster-node-tuning-operator  tuned-xx8dc                      10m (0%)          0 (0%)         50Mi (0%)      0 (0%)         4h5m
openshift-dns                              dns-default-4bh75                     65m (4%)    0 (0%)         131Mi (1%)     0 (0%)         4h5m
openshift-image-registry                   node-ca-rcnrc                         10m (0%)    0 (0%)         10Mi (0%)      0 (0%)         4h5m
openshift-ingress-canary                   ingress-canary-q5dpl                  10m (0%)    0 (0%)         20Mi (0%)      0 (0%)         4h3m
openshift-machine-config-operator          machine-config-daemon-xl9tx           40m (2%)    0 (0%)         100Mi (1%)     0 (0%)         4h5m
openshift-marketplace                      redhat-marketplace-c8sfx              10m (0%)    0 (0%)         50Mi (0%)      0 (0%)         104m
openshift-monitoring                       grafana-7874c696f6-v8498              5m (0%)   0 (0%)         120Mi (1%)     0 (0%)         3h59m
openshift-monitoring                       node-exporter-z4zk6                   9m (0%)   0 (0%)         210Mi (3%)     0 (0%)         4h5m
openshift-monitoring                       prometheus-k8s-1                      76m (5%)    0 (0%)         1204Mi (18%)   0 (0%)         3h59m
openshift-monitoring                       thanos-querier-8479f8f5cd-c6kvj       9m (0%)   0 (0%)         92Mi (1%)      0 (0%)         3h59m
openshift-multus                           multus-28dth                          10m (0%)    0 (0%)         150Mi (2%)     0 (0%)         4h5m
openshift-multus                           network-metrics-daemon-wnw5r          20m (1%)    0 (0%)         120Mi (1%)     0 (0%)         4h5m
openshift-network-diagnostics              network-check-target-5crw2            10m (0%)    0 (0%)         15Mi (0%)      0 (0%)         4h5m
openshift-sdn                              ovs-gx5g2                             15m (1%)    0 (0%)         400Mi (6%)     0 (0%)         4h5m
openshift-sdn                              sdn-7kr86                             110m (7%)    0 (0%)         220Mi (3%)     0 (0%)         4h5m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests       Limits
  --------                    --------       ------
  cpu                         439m (29%)     0 (0%)
  memory                      3042Mi (45%)  0 (0%)
  ephemeral-storage           0 (0%)         0 (0%)
  hugepages-1Gi               0 (0%)         0 (0%)
  hugepages-2Mi               0 (0%)         0 (0%)
  attachable-volumes-aws-ebs  0              0
Events:                       <none>

You can display memory and CPU usage statistics about nodes, which provide the runtime environments for containers.

%  oc adm top nodes
NAME                                         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
ip-10-0-135-85.us-west-2.compute.internal    271m           18%     2543Mi         38%
ip-10-0-155-129.us-west-2.compute.internal   617m           17%     5347Mi         37%
ip-10-0-164-249.us-west-2.compute.internal   213m           14%     3169Mi         47%
ip-10-0-189-65.us-west-2.compute.internal    536m           15%     5404Mi         37%
ip-10-0-193-29.us-west-2.compute.internal    678m           19%     7156Mi         48%
ip-10-0-214-229.us-west-2.compute.internal   321m           21%     2903Mi         43%

The screenshot below shows the AWS instances created by IPI for the OpenShift cluster.

Screenshot of AWS instances created by IPI for the OpenShift cluster

Add a c5.2xlarge instance to the OpenShift cluster

We will now see how to add an c5.2xlarge instance to the OpenShift cluster using the OpenShift CLI.

MachineSets are groups of machines. MachineSets are to machines as ReplicaSets are to Pods. If you need more machines or must scale them down, you work with MachineSet to meet your compute needs.

# Select the first machineset
% SOURCE_MACHINESET=$(oc get machineset -n openshift-machine-api -o name | head -n1)
# Reformat with jq, for better diff result.
% oc get -o json -n openshift-machine-api $SOURCE_MACHINESET  | jq -r > /tmp/source-machineset.json
% OLD_MACHINESET_NAME=$(jq '.metadata.name' -r /tmp/source-machineset.json )
% NEW_MACHINESET_NAME=${OLD_MACHINESET_NAME/worker/worker-comp}
# Change instanceType to the c5.2xlarge instance that we are adding, and delete some stuff
 % jq -r '.spec.template.spec.providerSpec.value.instanceType = "c5.2xlarge"
  | del(.metadata.selfLink)
  | del(.metadata.uid)
  | del(.metadata.creationTimestamp)
  | del(.metadata.resourceVersion)
  ' /tmp/source-machineset.json > /tmp/comp-machineset.json
# Change machineset name
% sed -i "s/$OLD_MACHINESET_NAME/$NEW_MACHINESET_NAME/g" /tmp/comp-machineset.json
# Check changes via diff
% diff -Nuar /tmp/source-machineset.json /tmp/comp-machineset.json
Create the machine set
% oc create -f /tmp/comp-machineset.json
machineset.machine.openshift.io/mytest-qd52c-comp-us-west-2a created

You can see that the new machineset “mytest-qd52c-comp-us-west-2a” is being created below. This will take a couple of minutes to be ready.

% oc get machineset -n openshift-machine-api
NAME                             DESIRED   CURRENT   READY   AVAILABLE   AGE
mytest-qd52c-comp-us-west-2a     1      1                         106s
mytest-qd52c-worker-us-west-2a   1     1     1      1            9h
mytest-qd52c-worker-us-west-2b   1     1     1      1            9h
mytest-qd52c-worker-us-west-2c   1     1     1      1            9h
mytest-qd52c-worker-us-west-2d   0     0                         9h
% oc get machineset -n openshift-machine-api
NAME                             DESIRED   CURRENT   READY   AVAILABLE   AGE
mytest-qd52c-comp-us-west-2a     1     1     1      1            4m44s
mytest-qd52c-worker-us-west-2a   1     1     1      1            9h
mytest-qd52c-worker-us-west-2b   1     1     1      1            9h
mytest-qd52c-worker-us-west-2c   1     1     1      1            9h
mytest-qd52c-worker-us-west-2d   0     0                         9h

We can see on the AWS console that a new c5.2xlarge instance has been created.

Screenshot of the AWS console showing a new c5 2x large instance has been created

% oc get nodes
NAME                                           STATUS   ROLES      AGE   VERSION
ip-10-0-128-195.us-west-2.compute.internal   Ready   worker   70s   v1.20.0+bafe72f
ip-10-0-135-85.us-west-2.compute.internal      Ready  worker   9h       v1.20.0+bafe72f
ip-10-0-155-129.us-west-2.compute.internal   Ready   master   9h       v1.20.0+bafe72f
ip-10-0-164-249.us-west-2.compute.internal   Ready   worker   9h       v1.20.0+bafe72f
ip-10-0-189-65.us-west-2.compute.internal      Ready  master   9h       v1.20.0+bafe72f
ip-10-0-193-29.us-west-2.compute.internal      Ready  master   9h       v1.20.0+bafe72f
ip-10-0-214-229.us-west-2.compute.internal   Ready   worker   9h       v1.20.0+bafe72f

NOTE: For HA you must create the above on at least two AZ’s in that region.

Deploy an application on the newly added node

Label the new node “intel=comp-optimized”

% oc label node ip-10-0-128-195.us-west-2.compute.internal intel=comp-optimized
node/ip-10-0-128-195.us-west-2.compute.internal labeled
%

Create a new project call myproject with node-selector set as “intel=comp-optimized”

% oc adm new-project myproject --node-selector "intel=comp-optimized"       

Created project myproject

Next, I deployed an Image for the health-demo application from the Quay (quay.io/mayurshetty/health-demo)

Screenshot of a deployed Image for the health-demo application from the Quay
We can see that the health-demo pod was started on the new c5.2xlarge instance.
% oc get pods -o wide
NAME                   READY   STATUS         RESTARTS   AGE         IP             NODE                                             NOMINATED NODE   READINESS GATES
health-demo-1-cxzkf    1/1     Running        0      2m45s   10.131.0.13   ip-10-0-128-195.us-west-2.compute.internal   <none>              <none>
health-demo-1-deploy   0/1    Completed   0          2m51s   10.131.0.12   ip-10-0-128-195.us-west-2.compute.internal  <none>               <none>
%

Conclusion

In this post we have seen the various AWS EC2 instance types available for the different workload categories. We also looked into how OpenShift gives us the flexibility to add EC2 instances with Intel Xeon Processors to an OpenShift cluster running on AWS, and on how we can deploy our application to run on the new instance.

Reference

Written by Mayur Shetty, Principal Solution Architect, Red Hat and Raghu Moorthy, Principal Engineer, Intel Inc.