Configuring NGINX Plus as an External Load Balancer for Red Hat OCP and Kubernetes

Original: https://www.nginx.com/blog/configuring-nginx-plus-external-load-balancer-red-hat-ocp-kubernetes/

In the world of container orchestration there are two names that we run into all the time: RedHat OpenShift Container Platform (OCP) and Kubernetes. OpenShift, as you probably know, uses Kubernetes underneath, as do many of the other container orchestration platforms. Routing external traffic into a Kubernetes or OpenShift environment has always been a little challenging, in two ways:

In this blog, I focus on how to solve the second problem using NGINX Plus in a way that is simple, efficient, and enables your App Dev teams to manage both the Ingress configuration inside Kubernetes and the external load balancer configuration outside. As a reference architecture to help you get started, I’ve created the nginx-lb-operator project in GitHub – the NGINX Load Balancer Operator (NGINX-LB-Operator) is an Ansible‑based Operator for NGINX Controller created using the Red Hat Operator Framework and SDK. NGINX-LB-Operator drives the declarative API of NGINX Controller to update the configuration of the external NGINX Plus load balancer when new services are added, Pods change, or deployments scale within the Kubernetes cluster.

Please note that NGINX-LB-Operator is not covered by your NGINX Plus or NGINX Controller support agreement. You can report bugs or request troubleshooting assistance on GitHub.

Kubernetes and NGINX Technologies – A Review

NGINX-LB-Operator relies on a number of Kubernetes and NGINX technologies, so I’m providing a quick review to get us all on the same page. If you’re already familiar with them, feel free to skip to The NGINX Load Balancer Operator.

Kubernetes Controllers and Operators

Kubernetes is an orchestration platform built around a loosely coupled central API. The API provides a collection of resource definitions, along with Controllers (which typically run as Pods inside the platform) to monitor and manage those resources. The Kubernetes API is extensible, and Operators (a type of Controller) can be used to extend the functionality of Kubernetes.

NGINX Ingress Controllers for Kubernetes

There are two main Ingress controller options for NGINX, and it can be a little confusing to tell them apart because the names in GitHub are so similar. We discussed this topic in detail in a previous blog, but here’s a quick review:

For a summary of the key differences between these three Ingress controller options, see our GitHub repository.

NGINX Controller

NGINX Controller is our cloud‑agnostic control plane for managing your NGINX Plus instances in multiple environments and leveraging critical insights into performance and error states. Its modules provide centralized configuration management for application delivery (load balancing) and API management. NGINX Controller can manage the configuration of NGINX Plus instances across a multitude of environments: physical, virtual, and cloud. It is built around an eventually consistent, declarative API and provides an app‑centric view of your apps and their components. It’s designed to easily interface with your CI/CD pipelines, abstract the infrastructure away from the code, and let developers get on with their jobs.

When it comes to Kubernetes, NGINX Controller can manage NGINX Plus instances deployed out front as a reverse proxy or API gateway. It doesn’t make sense for NGINX Controller to manage the NGINX Plus Ingress Controller itself, however; because the Ingress Controller performs the control‑loop function for a core Kubernetes resource (the Ingress), it needs to be managed using tools from the Kubernetes platform – either standard Ingress resources or NGINX Ingress resources.

External Load Balancers

The NGINX Plus Ingress Controller for Kubernetes is a great way to expose services inside Kubernetes to the outside world, but you often require an external load balancing layer to manage the traffic into Kubernetes nodes or clusters. If you’re running in a public cloud, the external load balancer can be NGINX Plus, F5 BIG-IP LTM Virtual Edition, or a cloud‑native solution. If you’re deploying on premises or in a private cloud, you can use NGINX Plus or a BIG-IP LTM (physical or virtual) appliance. I’m told there are other load balancers available, but I don’t believe it 😉 .

When it comes to managing your external load balancers, you can manage external NGINX Plus instances using the NGINX Controller directly. Its declarative API has been designed for the purpose of interfacing with your CI/CD pipeline, and you can deploy each of your application components using it. But what if your Ingress layer is scalable, you use dynamically assigned Kubernetes NodePorts, or your OpenShift Routes might change?

In cases like these, you probably want to merge the external load balancer configuration with Kubernetes state, and drive the NGINX Controller API through a Kubernetes Operator. The diagram shows a sample deployment that includes just such an operator (NGINX-LB-Operator) for managing the external load balancer, and highlights the differences between the NGINX Plus Ingress Controller and NGINX Controller.

where:

In this topology, the custom resources contain the desired state of the external load balancer and set the upstream (workload group) to be the NGINX Plus Ingress Controller. NGINX-LB-Operator collects information on the Ingress Pods and merges that information with the desired state before sending it onto the NGINX Controller API.

The NGINX Load Balancer Operator

Writing an Operator for Kubernetes might seem like a daunting task at first, but Red Hat and the Kubernetes open source community maintain the Operator Framework, which makes the task relatively easy. The Operator SDK enables anyone to create a Kubernetes Operator using Go, Ansible, or Helm. At F5, we already publish Ansible collections for many of our products, including the certified collection for NGINX Controller, so building an Operator to manage external NGINX Plus instances and interface with NGINX Controller is quite straightforward.

I used the Operator SDK to create the NGINX Load Balancer Operator, NGINX-LB-Operator, which can be deployed with a Namespace or Cluster Scope and watches for a handful of custom resources. The custom resources map directly onto NGINX Controller objects (Certificate, Gateway, Application, and Component) and so represent NGINX Controller’s application‑centric model directly in Kubernetes. The custom resources configured in Kubernetes are picked up by NGINX-LB-Operator, which then creates equivalent resources in NGINX Controller.

NGINX-LB-Operator enables you to manage configuration of an external NGINX Plus instance using NGINX Controller’s declarative API. Because NGINX Controller is managing the external instance, you get the added benefits of monitoring and alerting, and the deep application insights which NGINX Controller provides.

This diagram illustrates how:

  1. You create custom resources in the project namespace which are sent to the Kubernetes API.
  2. NGINX-LB-Operator sees the newly configured resources and collects the desired state of the component and the deployment information from the Ingress controller in the Ingress namespace.
  3. A merged configuration from your definition and current state of the Ingress controller is sent to NGINX Controller.
  4. The configuration is delivered to the requested NGINX Plus instances and NGINX Controller begins collecting metrics for the new application.

Detailed deployment instructions and a sample application are provided on GitHub. If you don’t like role play or you came here for the TL;DR version, head there now.

A Sample Deployment

So let’s role play. I’ll be Susan and you can be Dave.

As Dave, you run a line of business at your favorite imaginary conglomerate. You’re down with the kids, and have your finger on the pulse, etc., so you deploy all of your applications and microservices on OpenShift and for Ingress you use the NGINX Plus Ingress Controller for Kubernetes. All of your applications are deployed as OpenShift projects (namespaces) and the NGINX Plus Ingress Controller runs in its own Ingress namespace.

You were never happy with the features available in the default Ingress specification and always thought ConfigMaps and Annotations were a bit clunky. This is why you were over the moon when NGINX announced that the NGINX Plus Ingress Controller was going to start supporting its own CRDs. Today your application developers use the VirtualServer and VirtualServerRoutes resources to manage deployment of applications to the NGINX Plus Ingress Controller and to configure the internal routing and error handling within OpenShift.

Sometimes you even expose non‑HTTP services, all thanks to the TransportServer custom resources also available with the NGINX Plus Ingress Controller. Developers can define the custom resources in their own project namespaces which are then picked up by NGINX Plus Ingress Controller and immediately applied. It’s awesome, but you wish it were possible to manage the external network load balancer at the edge of your OpenShift cluster just as easily. The times when you need to scale the Ingress layer always cause your lumbago to play up.

It’s Saturday night and you should be at the disco, but yesterday you had to scale the Ingress layer again and now you have a pain in your lower back. Ping! In a cloud of smoke your fairy godmother Susan appears.

“Hello, Dave,” she says.

“Who are you? Look what you’ve done to my Persian carpet,” you reply.

Ignoring your attitude, Susan proceeds to tell you about NGINX-LB-Operator, now available on GitHub. She explains that with an NGINX Plus cluster at the edge of OpenShift and NGINX Controller to manage it from an application‑centric perspective, you can create custom resources which define how to configure the NGINX Plus load balancer.

The NGINX-LB-Operator watches for these resources and uses them to send the application‑centric configuration to NGINX Controller. In turn, NGINX Controller generates the required NGINX Plus configuration and pushes it out to the external NGINX Plus load balancer.

Your end users get immediate access to your applications, and you get control over changes which require modification to the external NGINX Plus load balancer!

NGINX Controller collects metrics from the external NGINX Plus load balancer and presents them to you from the same application‑centric perspective you already enjoy. And next time you scale the NGNIX Plus Ingress layer, NGINX-LB-Operator automatically updates the NGINX Controller and external NGINX Plus load balancer for you. No more back pain!

Conclusion

Kubernetes is a platform built to manage containerized applications. NGINX Controller provides an application‑centric model for thinking about and managing application load balancing. NGINX-LB-Operator combines the two and enables you to manage the full stack end-to-end without needing to worry about any underlying infrastructure. Head on over to GitHub for more technical information about NGINX-LB-Operator and a complete sample walk‑through.

Learn more about our solutions:

Retrieved by Nick Shadrin from nginx.com website.