IBM is going pretty competitive with its offerrings against like the likes of AWS and Google Cloud. They are offering a pretty good startup credits. You can find the link to get the benefit here https://developer.ibm.com/startups/

Moving to IBM is not that easy though, not because of their offering, but because there is very less “external” documentation available. A lot of the times you could be stuck figuring out what will make your application work.

I am writing thing articles that contains a lot of “Gotchas” right from setting up Toolchain and Continuous deployment to Having your application run on Kubernetes with TLS Passthrough and Termination.

Now this deployment is specific to my setup but I am sure it should apply to most of you.

My tasks were:-

Move my application from AWS beanstalk to Kubernetes and Docker in IBM
Ensure HTTPS
Implement Continuous Delivery

Here is my setup..

Two services written in Go
a) “Gateway” (Should run HTTPS with Termination at NGINX, secondly it forwards the requests to Gateway my Java App named “books-erp”)
b) “x509 Gateway” (Should run HTTPS with PASSTHROUGH as I am authenticating certs myself, it further connects with other services inside the cluster)
Backend Application in Java
Does not need HTTPS, connects with Database in Postgres.

Its not complex, since I did not have go services before and neither the application containerized. So the task ahead was steep. Let’s get started.

STEP 1: Create IBM Account and Starting Kubernetes Cluster.

I would recommend you chose the configuration “above” my VM config if not the same. I chose the following

3 worker nodes
16GB 4 CPU VM
Kubernetes 1.17.
Loaction Chennai (This was a mistake)

Another important thing when we create kubernetes is to create a namespace.

So I did an created a namespace “development”, but it did not work as IBM would say it is already in use. Huh, pretty confusing.

Gotchas

Since Chennai was my obvious choice as most of our customers are in India, however it seems like in IBM with “Indian Billing” you cannot choose your cluster location as Chennai.
Custom Namespaces are Unique in IBM network across all accounts, so its much better to either use “default” or create a namespace with something super unique to you

Step 2: Create Dockerfile

So I went ahead and wrote Dockefile for each of my applications, pretty standard stuff. Few recommendations.

Use a multi layer docker build process.
Make sure that the Build Step is part of your Dockerfile instead of Makefile/local gradle.
Try to ensure that your Image builds on local machine before you think about moving to Toolchain in IBM.
Dont be afraid to use build-args or env variables, you can easily configure that in Toolchain.

Step 3: Write YAML and Configure Toolchain

There are a one assumption here. I am using Deploy with Kubernetes Toolchain, with Github,

Link: https://cloud.ibm.com/devops/setup/deploy?repository=https%3A%2F%2Fgithub.com%2Fopen-toolchain%2Fsecure-kube-toolchain

Move to each step and configure the repo branch. There are few important variables.

Build Step

Make sure the repository branch is correct in the INPUT Tab.
You may chose to remove the Unit test Job in the JOBS Tab
In Environment Props, Make sure the IMAGE_NAME is the name of the app (its not necessary, but easier to remember)

Containerize Step

Make sure that the Image Name property in the JOBS Tab matched with the environment prop in the build step.
You may like to supply Build Args by creating a new property with name
EXTRA_BUILD_ARGS value with complete property as you would do when building docker image. example — build-arg APP=erp
You may want to modify the Dockerfile if its named differently.

Deploy Step

In the Environment Props Tab make sure the CLUSTER_NAMESPACE is the namespace with the unique namespace name you created or “default” if you want to use the default namespace.
DEPLOYMENT_FILE should be the path of the custom deployment and service file you created, this step will automatically modify the Image key in your deployment file.

Step 4: Database Connection

With the application getting deployed on Kubernetes, next step is to connect the application with the database.

I use both Mongo and Postgresql so I created the Database for PostgreSQL and MongoDB in the same region as my kubernetes cluster.

Gotcha: IBM requires you to use SSL Verify full mode to connect with the database otherwise it will reject the connection request.

Another Tip: You can also do SSL without verification by adding the following to your connection string.

Postgres Specific String: ?ssl=true&sslmode=require&sslfactory=org.postgresql.ssl.NonValidatingFactory&

Note: "&" at the end is intentional!

Follow the steps https://cloud.ibm.com/docs/services/databases-for-postgresql?topic=cloud-databases-tutorial-k8s-app&locale=de, just make sure the namespace is correct to your kubernetes cluster and you bind the cluster with the service.

Binding will give you the username and the password to access the database, you can also use admin credentials if you like. You can find the credentials in the service credentials section of the databases resource list.

With the detail, you should be able to connect to your database from within your kubernetes cluster app.

Step 5: Ingress and HTTPS Setup

With the application running on ClusterIP and database connected properly, its time to connect the services together and expose the endpoints to the public.

So I did a lot of research around different ingress services and there were a couple of options in front of me since I was using IBM.

Option 1: Use IBM ALB (default)
This option is really very easy and IBM already sets up the cluster and load balancer with an endpoint for you, all you need is to write an Ingress application and connect your application. You can find the details as well as installation information here. https://cloud.ibm.com/docs/containers?topic=containers-ingress-about

However this option did not work for me and I will explain the reason in Option 2.

Option 2: Use Managed/Self Install Istio Istio is great and much ahead of the curve when it comes to routing your traffic and doing other fancy stuff which you get out of the box, you can read more about istio here. https://istio.io/

Moreover you can follow this medium article to get a really good walkthrough of istio. https://medium.com/google-cloud/back-to-microservices-with-istio-p1-827c872daa53

So the reason I went with istio was, I had to expose my application in a unique way. I have 2 different endpoints to expose.

Gateway (this needs TLS termination support)
x509Gateway (this needs TLS passthrough support)

This is not possible to in ALB without creating my own Load Balancer.
Note: You can do that if you are interested by checking this out. https://www.ibm.com/cloud/blog/bring-your-own-alb-dns-with-health-checks-and-ssl-certificates-beta)

Another thing I really liked was kiali, its a tool that allows you to visualise your entire traffic flow, and much more, if you havent read the medium article above, I would recommend you do that now. Use the guide to setup your files, in case you are stuck, please comment on this post and I would love to help you out. I am not mentioning my setup as its super specific to my needs.

Gotchas

You will notice you are not able to login to kiali, the reason is you do not have a secret setup for it. Use the following command to do it.
kubectl create secret generic kiali -n istio-system --from-literal=username=admin --from-literal=passphrase=admin
Once you login to kiali you will see some errors, like you need named ports, or sidecards not injected, or you have duplicate host names. Please have a look at some of the requirements that istio specifies. https://istio.io/docs/ops/deployment/requirements/
In case you are making an http request from within your application to another app inside your cluster, I am guessing you will be using the service name to make a request. That’s right, however you will need to make one more change, make sure you “rewrite” your host header to the current service. (I can get into details if you are facing an issue with visualising your traffic in kiali)
Make sure you have the match.uri.prefix specified in VirtualService. I found that istio only considers / as path, and ignores everything else, so in my case I had to explicitly put the config to make sure the routing works. This is how it looks for me.

http:
  - match:
    - uri:
        prefix: /
    route:
      - destination:
          host: gateway
          port:
            number: 5000

Adding external service (database) to istio

Since I use mongo and postgres both, its fairly easy to connect mongo with istio. Have a look here for documentation https://istio.io/blog/2018/egress-mongo/

My config looks like this.

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: mongo
spec:
  hosts:
  - xxxxxxx.xxxxx.xxxx
  ports:
  - number: 12701
    name: tls
    protocol: TLS
  resolution: DNS

Istio does not support STARTTLS so you cannot connect postgres to istio unless postgres is part of your kubernetes cluster. It will show up as Passthrough Cluster and you gotta live with it.

Setting up Cert Manager

In order to setup https and sds, you need to have cert manager installed in your kubernetes cluster, follow the instructions here https://cert-manager.io/docs/installation/kubernetes/

Once you install you need to create a ClusterIssuer and Certificate.

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager # THIS IS IMPORTANT. 
type: Opaque
stringData:
  api-token: REDACTED
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: 'abc@gmail.com'
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - dns01: # I tried to setup dns01 later (check my comment below)
        cloudflare:
          email: cloudflare-email@gmail.com
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-token
    - http01:
        ingress:
          class: istio---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: ingress-cert
  namespace: istio-system
spec:
  secretName: ingress-cert
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: xxxxxxxx.com # Make sure this name is also the part of your domains and dnsnames below. 
  dnsNames:
    - xxxxxxx.com
    - ppppppp.com

When you try to apply this you might see a validation error so make sure you apply this config with the following command.

kubectl apply -f file.yaml --validate=false

Some debugging tips to Issue Certificates (Cert-Manager)

DNS01
It would be good to note that dns01 should be the easiest way to issue certificates. However you have to look at the following options.

Make sure you start with dns01 solution instead of http01, in case you did not and now making a change, be vary of the fact that Cert “Order” will still get issued with http01 challenge instead of dns01, in order to issue dns01, you need to delete your certificate and secret first.
Make sure that the “apiTokenSecretRef” is stored in the SAME namespace as cert-manager otherwise you will get errors like Secret Not found.

HTTP01
I started with HTTP01 thinking it would be the easiest of all, however with istio, the certificate issuer would be stuck with waiting and those things are difficult to resolve. Here are some debugging tips.

In case “kubectl describe cert ingress-cert -n istio-system” shows waiting for a long time you can debug with the following things to find the root cause.
Run “kubectl describe challenge -n istio-system” to see if it lists out challenges, in case a server is not resolving, you can just redeploy that server.
If there are no challenges found have a look at CertificateRequest object in istio-system, you will be able to see an error which you can solve.
You can also try to run “kubectl describe order -n istio-system” to see if the order is generated properly.

Adding TLS certificates

You need to create multiple gateway config for different types of exposures, in my case I created 3 gateways.

HTTP
TLS Passthrough
TLS Termination

My config is as below, the important thing to note is that in TLS Termination, your config needs to say “sds” instead of ingressgateway-certs/key.pem, if you follow the documentation here https://istio.io/docs/tasks/traffic-management/ingress/ingress-certmgr/, you will see that the manifest creates a k8singress deployment and it also later patches the file to add “sds” in the tls settings, however I would recommend you write the file like I mention below so that you have more control.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: http-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - xxxxxxxxx.com
        - ppppppppp.com---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: tls-passthrough
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: tls-passthrough
        protocol: HTTPS
      tls:
        mode: PASSTHROUGH
      hosts:
        - sssssssss.com
        - qqqqqqqqq.com
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: tls-termination
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: tls-termination
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: ingress-cert
        serverCertificate: sds
        privateKey: sds
      hosts:
        - wwwwwwwwww.com
        - aaaaaaaaaa.com

Gotcha: In case you are using managed istio by ibm, you will see that ingress-sds gets deleted automatically and is not reliable, that’s because SDS is not yet supported by managed istio, as of writing this article, istio 1.5 supports SDS by default and is yet to be taken live on managed istio on IBM.

In case you really need sds (which is the best practice), I would recommend you install istio manually via istioctl. I also generated a manifest for myself with some modifications (deleting the gateway config it generates, since I manually cread the config above and this also removed the multiple hosts error that you will see in kiali)

With the above steps done, you should be able to finally see your application working without any issues, it was a difficult setup with a lot of gotchas, but the more I worked around with the setup, the more I started liking IBM and its documentation, its consistent and something you can reply upon.

Another great thing about IBM is its slack community for IBM Cloud Services(https://ibm-cloud-success.slack.com/) and Cloud Devops Organisations (https://ic-devops-slack-invite.us-south.devops.cloud.ibm.com/)

If you are stuck anywhere (since I did not give complete instructions, rather just the issues I faced), please comment below and I can help out.

Credits

I would like to thank a few people who helped me tremendously to get the setup working

Eric Jodet (IBM Cloud Devops)
Benoit Jaunin (IBM Cloud Devops)
Greg Hanson (IBM Cloud Services, Managed Istio and Knative)
Baker Pratt (IBM Cloud Services)

Harsh Maur