Overview​

RSA delivers G&L for Kubernetes as two files: a container image with WildFly and the application installed, and a kit whose script, createYamlFiles.sh, writes the Kubernetes manifests and a run.sh that applies them (Installation Guide, Ch. 4).

The customer provides everything else: the cluster, the Oracle 19c database, a container registry, NFS storage, a load balancer (or NodePort access) and the firewall rules between them.

Scope: a new installation, as a single node (the master, which RSA also calls the domain node) or as a cluster of the master plus host nodes. The guide covers what is specific to Kubernetes. Tasks that are the same for every G&L deployment, such as preparing the database, the post-install database steps and backups, are listed with a link to RSA's guide. AFX and remote agents run on separate servers and are not covered.

Architecture​

Each G&L node runs as one pod. The kit gives each pod an external Service (LoadBalancer or NodePort) and an internal ClusterIP Service for the other G&L pods.

The schema is in Oracle, and the configuration and encryption keys are on NFS, so a pod can be recreated on another worker without data loss. Application logs are the exception: they are inside the pod (see Day-2 operations).

Network paths​

Port numbers for G&L are from the Installation Guide, Ch. 2, Port Requirements.

Prerequisites​

Confirm every row before Step 1.

Media

Workstation tools: kubectl, skopeo and curl.

Placeholders used in this guide

Installation flow​

Each step names where to run it and ends with a Check. Do not continue until the check passes.

Time required​

Measured in a test deployment on a clean cluster and a new database: two workers with 4 vCPU and 9 GB each, a 6144 MB heap, and the registry on the same local network as the cluster. Other hardware and networks give different times. Steps not listed are manual.

Step 1: Confirm the Oracle database​

Run on: each worker that can run G&L.

The database is prepared as for any G&L deployment, by RSA's database-only installer or by the DBA, following the Database Setup and Management Guide, Ch. 2, and the database grants in the P11 HF01 Release Notes. G&L creates its objects in AVUSER on its first start.

Collect from the DBA:

G&L connects to the database from its pod, so from whichever worker runs the pod, not from the control-plane node.

Check: each worker reaches the listener:

timeout 5 bash -c 'exec 3<>/dev/tcp/<db-host>/1555' && echo reachable
# reachable

Step 2: Stage the kit​

Run on: the control-plane node, as a user with sudo. The kit's README requires the script to run on the master node.

mkdir -p ~/gl-k8s && cd ~/gl-k8s
unzip /path/to/ACM_Kubernetes_V8.0.0.zip
chmod 755 createYamlFiles.sh

Check: createYamlFiles.sh is executable, and sudo kubectl get nodes shows every node Ready. The generated run.sh uses sudo kubectl, so root needs a working kubeconfig.

Step 3: Prepare the shared volumes​

Run on: the NFS server, then every worker, as root.

Why every worker needs the mount​

The kit does not use a StorageClass. It writes hostPath PersistentVolumes that point at directories on the node, and claims that bind to them:

A hostPath volume is the directory on whichever worker runs the pod. Without the NFS mount, the pod gets an empty local directory and cannot find its configuration or keys.

NFS server​

The directory names are RSA's examples (Installation Guide, Ch. 2, Kubernetes Prerequisites).

dnf install -y nfs-utils                       # Ubuntu: apt install nfs-kernel-server
mkdir -p /var/nfs/general
echo '/var/nfs/general <worker-1-ip>(rw,sync,no_subtree_check) <worker-2-ip>(rw,sync,no_subtree_check)' >> /etc/exports
systemctl enable --now nfs-server
exportfs -ra

cd /var/nfs/general
mkdir dataiglconfig76 dataiglkeystore76 dataiglsecurity76
mkdir dataiglconfig761                         # cluster: one config directory per host
chown 1000:1000 dataiglconfig76 dataiglkeystore76 dataiglsecurity76 dataiglconfig761
chmod 755 dataiglconfig76 dataiglkeystore76 dataiglconfig761
chmod 700 dataiglsecurity76

In a cluster, each host has its own config directory and shares the master's keystore and security directories.

Every worker, and the control-plane node​

createYamlFiles.sh (Step 6) writes each node's Aveksa_System.cfg into the config directory on the node where it runs, so that node needs the same mount.

dnf install -y nfs-utils                       # Ubuntu: apt install nfs-common
mkdir -p /var/nfs/general
echo '<nfs-server>:/var/nfs/general /var/nfs/general nfs defaults,_netdev 0 0' >> /etc/fstab
mount /var/nfs/general

Check: on every worker and the control-plane node, the share is mounted, owned by uid 1000, and the security directory is mode 700:

mountpoint /var/nfs/general
# /var/nfs/general is a mountpoint

ls -ldn /var/nfs/general/dataigl*
# drwxr-xr-x. 2 1000 1000 31 Oct  6 03:13 /var/nfs/general/dataiglconfig76
# drwxr-xr-x. 2 1000 1000 31 Oct  5 23:07 /var/nfs/general/dataiglconfig761
# drwxr-xr-x. 2 1000 1000 91 Oct  6 09:36 /var/nfs/general/dataiglkeystore76
# drwx------. 2 1000 1000 36 Oct  5 22:52 /var/nfs/general/dataiglsecurity76

Step 4: Push the G&L image​

Run on: the admin workstation.

RSA's Kubernetes chapter loads the image with docker load on each node. Showstopper on current clusters: Kubernetes 1.24 removed dockershim, and containerd does not see images in Docker's store (Dockershim removal FAQ). Workaround: push the image to the registry, which the kit's README also allows ("one can have a private registry with the image uploaded in it"):

Read the tag recorded in the tar, then copy the tar to the registry with skopeo, which needs no container engine:

tar -xOf ACM_Container_VP11_HF01_8.0.0.189610.tar manifest.json      # RepoTags: identity/rsaigl:8.0.0.189610

skopeo copy --dest-creds '<push-user>:<password>' --dest-cert-dir <dir-with-ca.crt> \
  docker-archive:ACM_Container_VP11_HF01_8.0.0.189610.tar \
  docker://<registry>/rsa/acm:8.0.0.189610

--dest-cert-dir is needed only for a private CA. That CA must also be in every node's operating system trust store (for example /etc/pki/ca-trust/source/anchors/, then update-ca-trust), or pulls fail with x509: certificate signed by unknown authority.

Check:

curl -u <user> https://<registry>/v2/rsa/acm/tags/list       # {"name":"rsa/acm","tags":["8.0.0.189610"]}

Step 5: Create the namespace and pull secret​

Run on: the control-plane node.

The kit's manifests reference no pull secret. Create one in the namespace (Pull an image from a private registry):

kubectl create namespace rsaigl
kubectl create secret docker-registry gl-registry -n rsaigl \
  --docker-server=<registry> --docker-username=<pull-user> --docker-password='<password>'

Test the pull on each G&L worker. The pod also prints the user the image runs as, and leaves the 4 GB image cached on the worker:

apiVersion: v1
kind: Pod
metadata:
  name: gl-pull-test
  namespace: rsaigl
spec:
  nodeName: <worker-1>
  restartPolicy: Never
  imagePullSecrets:
  - name: gl-registry
  containers:
  - name: probe
    image: <registry>/rsa/acm:8.0.0.189610
    imagePullPolicy: IfNotPresent
    command: ["sh", "-c", "id"]

kubectl apply -f gl-pull-test.yaml
kubectl logs gl-pull-test -n rsaigl          # uid=1000(jboss) gid=1000(jboss) groups=1000(jboss)
kubectl delete pod gl-pull-test -n rsaigl

Repeat with nodeName: <worker-2> in a cluster.

Check: the pod completes on every G&L worker and prints uid 1000. If the uid differs, use it for the chown commands in Steps 3 and 7.

Step 6: Generate the manifests​

Run on: the control-plane node, in ~/gl-k8s.

createYamlFiles.sh writes the manifests and each node's Aveksa_System.cfg, including the cluster parameters, into the config paths it is given on this node. It writes the file only when the config directory has none; an existing file is used as it is (Installation Guide, Ch. 4). Leave the config directories empty, or the nodes will not form a cluster. The directories belong to uid 1000 (Step 3), so run the script with sudo.

ls /var/nfs/general/dataiglconfig76 /var/nfs/general/dataiglconfig761    # no Aveksa_System.cfg
sudo ./createYamlFiles.sh

Prompts in the order the 8.0 kit asks them. Enter accepts the default in brackets.

The Installation Guide's prompt list omits prompts 11, 12, 16 and 17.

By default, this would create manifest for domain node. How many IGL application host's nodes are you planning to deploy in the Kubernetes Cluster : 1

Provide the image name : <registry>/rsa/acm:8.0.0.189610

Enter the Remote Database IP/Hostname : <db-host>
Provided remote database is :  <db-host>

Enter the AVUSER password :
Confirm the AVUSER password :

Enter the IGL application master host name [rsaiglmaster]:
The provided IGL application master hostname is :  rsaiglmaster

Enter the IGL application master service name [rsaiglsvcmaster]:
The entered IGL application master service name is :  rsaiglsvcmaster

Enter master's config path : /var/nfs/general/dataiglconfig76
Provided master's config path is :  /var/nfs/general/dataiglconfig76

Enter master's keystore path : /var/nfs/general/dataiglkeystore76
Provided master's keystore path is :  /var/nfs/general/dataiglkeystore76

Enter master's security path : /var/nfs/general/dataiglsecurity76
The provided master's security path is :  /var/nfs/general/dataiglsecurity76

Enter the namespace in which the rsaigl components to be deployed in the cluster [rsaigl]:
The provided namespace is :  rsaigl

Enter the LoadBalancer Hostname/IP required to access the application (Leave it empty if the value is unknown and edit it later) : <lb-ip>

Enter the port used to access the application (Leave it empty if the value is unknown and edit it later) : 8443

Enter the service Type for the Pods [NodePort/LoadBalancer] : LoadBalancer
Provided service type is :  LoadBalancer

### Creating the Aveksa_System.cfg file in the master config directory ###
----------------------------------------------------------------
The entered Pod Service type is :  LoadBalancer
----------------------------------------------------------------
Creating run.sh file
----------------------------------------------------------------
Number of IGL application hosts you selected to deploy in the K8s cluster is : 1

Enter the IGL application host1 hostname [host1]:
Provided host1 hostname is : host1

Enter the host1 config path : /var/nfs/general/dataiglconfig761
Provided host1 config path is :  /var/nfs/general/dataiglconfig761

Enter the LoadBalancer Hostname/IP required to access the application (Leave it empty if the value is unknown and edit it later) : <lb-ip-host1>

Enter the port used to access the application (Leave it empty if the value is unknown and edit it later) : 8443

### Creating Aveksa_System.cfg file under the given config path ie., /var/nfs/general/dataiglconfig761  ###

### Creating of Aveksa_System.cfg file for host1 is completed  ###

The cluster parameters the kit writes:

Generated files:

Check: the script prints "### Creating the Aveksa_System.cfg file in the master config directory ###" (and, in a cluster, the same for host1), and each node's file has its role:

sudo grep -H NODE_TYPE /var/nfs/general/dataiglconfig76*/Aveksa_System.cfg
# /var/nfs/general/dataiglconfig761/Aveksa_System.cfg:NODE_TYPE=Host
# /var/nfs/general/dataiglconfig76/Aveksa_System.cfg:NODE_TYPE=Domain

Step 7: Complete Aveksa_System.cfg​

Run on: the NFS server, as root.

The kit's file holds the database host, the AVUSER password, the load balancer address and the cluster parameters. Add the keys it has no prompt for to each node's file (Installation Guide, Ch. 2, Verify Docker Container Requirements):

for d in /var/nfs/general/dataiglconfig76 /var/nfs/general/dataiglconfig761; do
cat >> $d/Aveksa_System.cfg <<'EOF'
REMOTE_ORACLE_PORT=1555
REMOTE_ORACLE_DATABASE=AVDB
AVEKSA_USER=avuser
AVEKSA_REPORTS_USER=avdwuser
AVEKSA_REPORTS_PASS=<avdwuser-password>
AVEKSA_CSUSER=avcsuser
AVEKSA_CSUSER_PASS=<avcsuser-password>
AVEKSA_PUBLIC_DB_USER=ACMDB
AVEKSA_PUBLIC_DB_PASS=<acmdb-password>
AVEKSA_AVPERF_USER=perfstat
AVEKSA_AVPERF_PASS=<perfstat-password>
WILDFLY_HEAP_MEM=12288
AMAZON_RDS=N
EOF
chown 1000:1000 $d/Aveksa_System.cfg
chmod 777 $d/Aveksa_System.cfg
done

sed -i 's/^AVEKSA_PASS=.*/AVEKSA_PASS=<avuser-password>/' /var/nfs/general/dataiglconfig761/Aveksa_System.cfg

Check: in each config directory, ls -ln Aveksa_System.cfg shows 1000 1000 and -rwxrwxrwx, cut -d= -f1 Aveksa_System.cfg lists 24 keys, and NODE_TYPE is still present.

Step 8: Correct the manifests​

Run on: the control-plane node, in ~/gl-k8s.

The kit's output does not deploy a working G&L on Kubernetes as generated. The Installation Guide allows the generated YAML to be edited (Ch. 4). The changes are grouped into kit defects that stop the deployment, an RSA requirement the kit has no prompt for, and recommendations. Keep the originals first:

for f in *.yaml; do cp -n "$f" "$f.orig"; done

Showstopper 1: bind each claim to its volume​

All kit volumes are identical apart from their path (storageClassName: manual, 8Gi, ReadWriteMany), and the claims name no volume, so Kubernetes binds a claim to any free volume. In a cluster deployment, configmaster-claim was bound to a keystore volume and securitymaster-claim to a config volume. Workaround: reserve each volume with volumeName (Reserving a PersistentVolume):

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: configmaster-claim
spec:
  storageClassName: manual
  volumeName: configmaster
  accessModes:
    - ReadWriteMany

Showstopper 2: pull secret​

Workaround: add the secret from Step 5 to each Deployment, under spec.template.spec:

      imagePullSecrets:
      - name: gl-registry

Showstopper 3 (cluster): host names​

CLUSTER_NODES_IP lists the nodes by host name, and each node must resolve the others' names to their pod addresses: the Installation Guide's fix copies each container's address into every container's /etc/hosts (Ch. 4, Common Problem). Kubernetes rewrites /etc/hosts at each pod start and gives a recreated pod a new address. A name that resolves to a Service's ClusterIP is not enough: discovery through it finds no members. Workaround: a headless Service per host name, which resolves the name to the pod's own address in cluster DNS (Headless Services). Add one to each internal Service file:

# appended to service_internal_master.yaml; service_internal_host1.yaml gets
# the same with name: host1 and selector name: host1
---
apiVersion: v1
kind: Service
metadata:
  name: rsaiglmaster
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    name: rsaiglmaster
  ports:
  - name: jgroups-tcp
    port: 7600
  - name: jgroups-tcp-fd
    port: 57600

Showstopper 4 (cluster): member list before start​

The image's start script, /home/oracle/execute.sh, writes CLUSTER_NODES_IP into WildFly's domain.xml only after WildFly has started, and does not reload it. Every Kubernetes pod starts from the image, so the servers always run with the image's 127.0.0.1[7600]. Workaround: set the same value before execute.sh runs, in the master's G&L container. The master is the domain controller and passes the setting to every host.

# master.yaml, G&L container
        command: ["/bin/sh", "-c"]
        args:
        - |
          f=/home/oracle/config/${CONFIG_FILE:-Aveksa_System.cfg}
          v=$(sed -n "s/^CLUSTER_NODES_IP=//p" "$f" | sed -E "s/([^,]+)/\1[7600]/g")
          [ -n "$v" ] && sed -i -E "s|(name=\"jboss.cluster.tcp.initial_hosts\"[[:space:]]+value=\")[^\"]*\"|\1$v\"|" /opt/jboss/wildfly/domain/configuration/domain.xml
          exec /home/oracle/execute.sh

The edit replaces only the value between the quotes. The image writes the element as value="127.0.0.1[7600]" />, with a space before />, so a pattern that expects "/> matches nothing and the servers still start with 127.0.0.1[7600].

RSA requirement 5: time zone​

The container must use the database's time zone; RSA's Docker procedure passes -e TZ when the database is not on America/New_York (Installation Guide, Ch. 5). The kit has no prompt for it. Add it to each G&L container:

        env:
        - name: TZ
          value: <tz>

Recommended 6 to 8​

      nodeSelector:
        kubernetes.io/hostname: <worker-1>      # 7: <worker-2> in host1.yaml
      containers:
      - name: rsaiglmaster
        imagePullPolicy: IfNotPresent          # 6

For 7, compare the 15Gi request with each worker's allocatable memory (kubectl get nodes -o custom-columns=NODE:.metadata.name,ALLOCATABLE:.status.allocatable.memory). A pod whose request exceeds every worker stays Pending (Resource management). Size the workers rather than lowering the request: a worker without enough memory kills WildFly as the heap grows (OOMKilled, exit code 137), possibly after the pod reported Ready.

For 8, the method depends on the load balancer. For MetalLB (MetalLB usage):

metadata:
  name: rsaiglsvcmaster
  annotations:
    metallb.universe.tf/loadBalancerIPs: <lb-ip>

Check: for f in *.yaml; do diff "$f.orig" "$f"; done shows only the changes above. For master.yaml in a cluster, the additions are:

>   volumeName: configmaster
>   volumeName: securitymaster
>   volumeName: keystoremaster
>         - name: TZ
>           value: <tz>
>         imagePullPolicy: IfNotPresent
>         command: ["/bin/sh", "-c"]
>         args:
>         - |
>           f=/home/oracle/config/${CONFIG_FILE:-Aveksa_System.cfg}
>           ...
>           exec /home/oracle/execute.sh
>       imagePullSecrets:
>       - name: gl-registry
>       nodeSelector:
>         kubernetes.io/hostname: <worker-1>

host1.yaml has the same changes without command and args, and each service_internal_*.yaml gains its headless Service.

Step 9: Deploy​

Run on: the control-plane node, in ~/gl-k8s.

cat run.sh
./run.sh
kubectl get pods -n rsaigl -o wide -w

• The first start creates the schema and is the slowest. Follow it with kubectl logs -f deploy/rsaiglmaster-deployment -n rsaigl.
• In a cluster, host1 stays Init:0/1 until the master answers on http://masterinternalsvc:8080; the init container checks every 300 seconds.
• The kit defines no readiness probe for the application, so Running does not mean G&L is serving. Step 10 confirms it.

A cluster deployment, watched with timestamps. host1 waits in its init container until the master serves, then starts:

TIME      EVENT      NAME                                       READY   STATUS            AGE     NODE
09:36:05  ADDED      host1-deployment-6599cf7489-n896j          0/1     Init:0/1          2m30s   <worker-2>
09:36:05  ADDED      rsaiglmaster-deployment-5bff7797df-tdm8d   1/1     Running           2m30s   <worker-1>
09:39:07  MODIFIED   host1-deployment-6599cf7489-n896j          0/1     PodInitializing   5m32s   <worker-2>
09:39:07  MODIFIED   host1-deployment-6599cf7489-n896j          1/1     Running           5m32s   <worker-2>

The master's container log shows the start sequence. The lines to look for:

Creating User for Domain Management
Created Domain Management user -  AveksaClusterAdmin
Fetching latest keystore password from the Database
Successfully created New Password Vault Keystore for wildfly server
Password Vault configured successfully...
Starting Domain-Controller Wildfly...
Wildfly Started Successfully
Deploying aveksa.ear...
aveksa.ear Deployed Successfully
Deploying aveksaWFArchitect.ear...
aveksaWFArchitect.ear Deployed Successfully
Updating passed DB credentials with encrypted values

Check: every pod is Running and 1/1, and each external Service has an EXTERNAL-IP:

kubectl get pods,svc -n rsaigl -o wide

NAME                                           READY   STATUS    RESTARTS   AGE     NODE
pod/host1-deployment-6599cf7489-n896j          1/1     Running   0          9m37s   <worker-2>
pod/rsaiglmaster-deployment-5bff7797df-tdm8d   1/1     Running   0          9m37s   <worker-1>

NAME                        TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)
service/host1               ClusterIP      None            <none>          7600/TCP,57600/TCP
service/internalsvc-host1   ClusterIP      10.99.67.190    <none>          1158/TCP,1155/TCP,...,9999/TCP,7600/TCP,57600/TCP
service/masterinternalsvc   ClusterIP      10.101.73.9     <none>          1158/TCP,1155/TCP,...,9999/TCP,7600/TCP,57600/TCP
service/rsaiglmaster        ClusterIP      None            <none>          7600/TCP,57600/TCP
service/rsaiglsvchost1      LoadBalancer   10.99.202.189   <lb-ip-host1>   8080:30351/TCP,8443:32738/TCP,8444:32726/TCP
service/rsaiglsvcmaster     LoadBalancer   10.96.158.145   <lb-ip>         8080:30401/TCP,8443:30872/TCP,8444:31124/TCP

rsaiglmaster and host1 are the headless Services added in Step 8.

Step 10: Verify the deployment​

Run on: the control-plane node, then a browser. Do not hand over until every check passes.

1. Pods are steady. RESTARTS is 0 on every pod.

kubectl get pods -n rsaigl
# NAME                                       READY   STATUS    RESTARTS   AGE
# host1-deployment-6599cf7489-n896j          1/1     Running   0          7m48s
# rsaiglmaster-deployment-5bff7797df-tdm8d   1/1     Running   0          7m48s

2. Every node answers on its own address. Each external Service returns 200 or 30x. A host starts after the master (its init check runs every 300 seconds), so retest a host's 000 before treating it as a failure.

curl -k -s -o /dev/null -w '%{http_code}\n' https://<lb-ip>:8443/aveksa/main          # 200
curl -k -s -o /dev/null -w '%{http_code}\n' https://<lb-ip-host1>:8443/aveksa/main    # 200

3. WildFly started and the application is deployed. The container log has the start lines and none of WFLYSRV0026, WFLYCTL0186 or rolled back. Ignore WFLYSRV0003: Could not index class warnings; they are normal.

for d in rsaiglmaster-deployment "host1-deployment -c host1"; do
  kubectl logs deploy/$d -n rsaigl | grep -E 'Starting (Domain|Host)-Controller|Wildfly Started Successfully|aveksa.ear Deployed Successfully|WFLYSRV0026|WFLYCTL0186|rolled back'
done
# Starting Domain-Controller Wildfly...
# Wildfly Started Successfully
# aveksa.ear Deployed Successfully
# Starting Host-Controller Wildfly...
# Wildfly Started Successfully

A host does not deploy aveksa.ear itself: the master, as domain controller, deploys it to every server in the domain.

4. Version. G&L's logs are inside the pod, under /opt/jboss/wildfly/domain/servers/<SERVER_NAME>/log (img-server-master on the master, img-server-1 on host1).

POD=$(kubectl get pod -n rsaigl -l name=rsaiglmaster -o name)
LOG=$(kubectl exec -n rsaigl $POD -- sh -c 'ls -d /opt/jboss/wildfly/domain/servers/*/log')
kubectl exec -n rsaigl $POD -- grep -m1 '^Version' $LOG/aveksaServerInfo.log
# Version 8.0.0.189610 P11_HF01

5. First logon. Open https://<lb-ip>:8443/aveksa/main and sign in as AveksaAdmin with aveksa123. G&L asks for a new password at once: at least eight characters, with upper case, lower case and a number (Installation Guide, Ch. 3).

6. Server Nodes. Go to Admin > System > Server Nodes. Each G&L node registers itself here when it starts, named <host name>-<SERVER_NAME>. Expect one row per node: exactly one System Operations Node, the rest General Node, all Healthy, with a recent heartbeat. A general node designated with Make Next SON shows (Next SON) after its role; see System Operations Node handover.

A node that was renamed or removed keeps its row, with its heartbeat stopped; delete it once it shows Shutdown or Unresponsive.

7. The nodes form one cluster (cluster only).

a. host1 joined the master's domain. host1 starts as a host controller and connects to the master; the master registers it.

kubectl logs deploy/host1-deployment -c host1 -n rsaigl | grep -E 'Starting (Domain|Host)-Controller'
# Starting Host-Controller Wildfly...

H=$(kubectl get pod -n rsaigl -l name=host1 -o name)
kubectl exec -n rsaigl $H -c host1 -- grep 'Connected to master host controller' /opt/jboss/wildfly/domain/log/host-controller.log
# WFLYHC0148: Connected to master host controller at remote://masterinternalsvc:9999

M=$(kubectl get pod -n rsaigl -l name=rsaiglmaster -o name)
kubectl exec -n rsaigl $M -- grep 'Registered remote slave host' /opt/jboss/wildfly/domain/log/host-controller.log
# WFLYHC0019: Registered remote slave host "slave1", ...

Starting Domain-Controller Wildfly on host1 means it is running as its own domain: see Step 6.

b. One cluster view with every node in it. Each server's latest view counts all the nodes, here (2) with both names:

for p in $M "$H -c host1"; do
  kubectl exec -n rsaigl $p -- sh -c 'grep ISPN000094 /opt/jboss/wildfly/domain/servers/*/log/server.log | tail -1'
done
# ISPN000094: Received new cluster view for channel ejb: [master:img-server-master|1] (2) [master:img-server-master, slave1:img-server-1]

A view of (1), or no members discovered ... creating cluster as coordinator, means the servers did not find each other: see Step 8, changes 3 and 4. The member list a server started with is in its JVM arguments:

kubectl exec -n rsaigl $M -- sh -c 'for d in /proc/[0-9]*; do tr "\0" "\n" < $d/cmdline 2>/dev/null | grep "^-Djboss.cluster.tcp.initial_hosts"; done'
# -Djboss.cluster.tcp.initial_hosts=rsaiglmaster[7600],host1[7600]

8. Pods are still steady. Repeat check 1. A worker short of memory can kill WildFly minutes after it reported Ready.

Step 11: Post-install tasks and backup​

Run on: the database server (the DBA), then the control-plane node and the NFS server.

Database tasks. These are the same for every G&L deployment and need objects that G&L creates on its first start: recompile AVUSER.Utilities_Pkg, confirm the additional AVUSER grants and the sys_perfstat migration row, and compare the database time zone with G&L's, as AVUSER (Database Setup and Management Guide, Ch. 2, Additional AVUSER Schema Permissions). If the time zones differ, the DBA corrects the database's; then restart the G&L pods (Stop and start).

Backup. Back up the database and the key encryption keys together; without the keys, encrypted data in the database cannot be recovered (Database Setup and Management Guide, Ch. 3). On Kubernetes the keys are in the security directory on NFS, and G&L is stopped and started with kubectl:

kubectl scale deployment --all --replicas=0 -n rsaigl                          # control-plane node
# database export by the DBA, per the Database Setup and Management Guide, Ch. 3
tar -C /var/nfs/general -czf /root/gl-kek-$(date +%F).tgz dataiglsecurity76  # NFS server
kubectl scale deployment rsaiglmaster-deployment --replicas=1 -n rsaigl        # then the hosts

Check: the DBA confirms the database tasks, and the database export and the key archive carry the same date and are stored together outside the cluster.

Day-2 operations​

Stop and start​

kubectl scale deployment --all --replicas=0 -n rsaigl
kubectl scale deployment rsaiglmaster-deployment --replicas=1 -n rsaigl
kubectl scale deployment host1-deployment --replicas=1 -n rsaigl     # cluster, after the master

Every start creates new containers, so confirm the cluster again with Step 10, check 7.

Logs​

The kit persists only config, keystore and security. WildFly and G&L logs are lost when a pod is deleted. Copy them first:

kubectl cp rsaigl/<pod>:/opt/jboss/wildfly/domain/servers/<SERVER_NAME>/log ./gl-logs-<pod>

Redeploy from scratch​

Deleting the namespace does not delete the PersistentVolumes, which are cluster-scoped, and it does delete the pull secret. Before running run.sh again:

kubectl delete namespace rsaigl
kubectl get pv                                    # the kit's volumes show Released
kubectl delete pv configmaster securitymaster keystoremaster config1 security1 keystore1

Then repeat Step 5 (namespace and pull secret) and Step 9. The data on NFS, including each Aveksa_System.cfg, is not affected: the volumes are hostPath with the Retain policy.

Newer image​

A container is not patched in place. Back up (Step 11) and push the new image (Step 4). In a cluster, scale the hosts to zero, change the master's image: and apply it, let the master migrate the schema alone, then start the hosts on the same image. A clustered deployment that is migrated or patched must run the System Operations Node without any other active node (Installation Guide, Ch. 7; Patch Installation Guide).

System Operations Node handover​

The first node started becomes the System Operations Node (SON). G&L does not fail over automatically: another node becomes SON only when an administrator designates it (Administrator's Guide, Ch. 2, Managing Server Cluster Nodes).

• Plan a successor: in Admin > System > Server Nodes, click Make Next SON on an active general node. It becomes SON the next time it starts after the current SON has been shut down.
• Replace a SON that is down: with the current SON shut down, click Make SON on the node to promote.
• Server Nodes can modify only inactive nodes (Shutdown or Unresponsive); use Delete to remove a node that is no longer part of the cluster.

RSA recommends directing user traffic to general nodes only. SON tasks such as collections may depend on files on the SON, so test them after a handover.

Troubleshooting​

Start with:

kubectl get events -n rsaigl --sort-by=.lastTimestamp | tail -20
kubectl describe pod <pod> -n rsaigl
kubectl logs <pod> -n rsaigl --previous

Pod Does Not Start​

Symptom: ErrImagePull: no basic auth credentials

• Root Cause: Image pull secret is missing in the namespace (for example, after recreating the namespace).
• Resolution: Re-create the pull secret as documented in Step 5.

Symptom: ErrImagePull: x509: certificate signed by unknown authority

• Root Cause: Private registry CA certificate is not trusted by the Kubernetes worker node.
• Resolution: Install and trust the registry CA certificate on each worker node per Step 4.

Symptom: ErrImagePull: not found

• Root Cause: Incorrect image registry reference, repository path, or tag.
• Resolution: Verify image name and tag in Step 4 and prompt 2 of Step 6.

Symptom: Pending: Insufficient memory

• Root Cause: The 15Gi memory request cannot be scheduled on any single worker node.
• Resolution: Scale worker node memory or adjust memory requests per Step 8, change 7.

Symptom: Pending: unbound immediate PersistentVolumeClaims; volumes Released

• Root Cause: Orphaned PersistentVolumes remain from a prior deployment.
• Resolution: Execute cleanup steps in Redeploy from scratch.

Symptom: Pod host1 remains in Init:0/1

• Root Cause: Master node has not answered within the probe window, or the nginx container image cannot be pulled.
• Resolution: Allow up to 300 seconds after master responds, or mirror the nginx image locally.

Pod Starts, Then Fails​

Symptom: Exit code 254 (no log produced)

• Root Cause: PersistentVolumeClaims are bound to the wrong volumes.
• Resolution: Verify volume bindings using kubectl get pvc -n rsaigl and apply Step 8, change 1.

Symptom: OOMKilled (exit code 137)

• Root Cause: Container memory consumption exceeded the node memory ceiling.
• Resolution: Increase worker memory or tune resource limits per Step 8, change 7.

Symptom: Permission denied on /home/oracle/security

• Root Cause: NFS directory is not owned by UID 1000 or file permissions are not 0700.
• Resolution: Correct directory permissions and ownership as shown in Step 3.

Symptom: Empty configuration or regenerated encryption keys

• Root Cause: Worker node is missing the shared NFS volume mount.
• Resolution: Confirm the NFS export is mounted across all worker nodes per Step 3.

Symptom: Database connection errors in log

• Root Cause: Incorrect database credentials in Aveksa_System.cfg (such as the default AVEKSA_PASS=secret), or the Oracle listener port is blocked from the worker node.
• Resolution: Validate database connectivity and credentials in Step 1 and Step 7.

G&L Runs But Has Irregularities​

Symptom: Node URL returns HTTP 000

• Root Cause: Target pod is not ready or is restarting.
• Resolution: Run kubectl describe pod <pod> -n rsaigl to verify pod health and probe status.

Symptom: Server Nodes UI lists fewer nodes than active pods

• Root Cause: Host pod is not registered with the domain master controller (host log lacks Connected to master host controller).
• Resolution: Confirm DOMAIN_MASTER in Step 6 and verify port 9999 connectivity on masterinternalsvc.

Symptom: Container log displays Starting Domain-Controller Wildfly on host

• Root Cause: Aveksa_System.cfg was not created by createYamlFiles.sh (missing NODE_TYPE=Host), running the host as an independent domain.
• Resolution: Regenerate manifests with NODE_TYPE=Host in Step 6.

Symptom: All nodes Healthy, but cluster view displays (1) member; server.log reports no members discovered

• Root Cause: Server started with -Djboss.cluster.tcp.initial_hosts=127.0.0.1[7600], or host names resolve to a Service IP instead of pod IP.
• Resolution: Apply Step 8, changes 3 and 4.

Symptom: EXTERNAL-IP remains <pending>

• Root Cause: Cluster lacks a load balancer controller or has exhausted external IP pool.
• Resolution: Provision a cluster load balancer (such as MetalLB) or change the Service type to NodePort.

Symptom: Timestamps are shifted

• Root Cause: Container timezone differs from the Oracle Database timezone.
• Resolution: Align container and database timezone configurations per Step 8, change 5, and Step 11.

Symptom: AFX cannot connect

• Root Cause: Port 8444 is unreachable on the external Service.
• Resolution: Open firewall port 8444 from AFX to the external Service IP.

References​

RSA Governance & Lifecycle 8.0

• RSA Governance & Lifecycle documentation (all guides and release notes, including 8.0 P11 HF01)
• Installation Guide 8.0: Ch. 2 requirements and ports, Ch. 4 Kubernetes, Ch. 5 Docker cluster parameters, Ch. 7 patching clusters
• Database Setup and Management Guide 8.0: Ch. 2 database configuration, Ch. 3 backup
• Upgrade and Migration Guide 8.0
• Patch Installation Guide 8.0: one node during a patch
• 8.0 Patch 11 HF01 Release Notes: database grants for P07 and later
• Administrator's Guide 8.0: Ch. 2, Managing Server Cluster Nodes

Kubernetes

• Persistent Volumes and Reserving a PersistentVolume
• hostPath volumes
• Pull an image from a private registry
• Headless Services and DNS for Services and Pods
• Assigning Pods to Nodes
• Resource management for Pods and containers
• Service
• Debug Pods
• Dockershim removal FAQ

Registry and load balancing

• skopeo copy
• Registry HTTP API V2
• MetalLB usage