G&L on a Kubernetes Cluster✓ Verified

October 2, 2026 · 41 min read

Creator

RSA Governance & Lifecycle (G&L) runs as containers on Kubernetes and connects to an Oracle database outside the cluster. This guide installs G&L 8.0 P11 HF01 (build 8.0.0.189610) on a cluster and database provided by the customer, as a single node or as a cluster of nodes, in eleven steps that each end with a check. It is written for installers new to G&L on Kubernetes.

Overview

RSA delivers G&L for Kubernetes as two files: a container image with WildFly and the application installed, and a kit whose script, createYamlFiles.sh, writes the Kubernetes manifests and a run.sh that applies them (Installation Guide, Ch. 4).

The customer provides everything else: the cluster, the Oracle 19c database, a container registry, NFS storage, a load balancer (or NodePort access) and the firewall rules between them.

Scope: a new installation, as a single node (the master, which RSA also calls the domain node) or as a cluster of the master plus host nodes. The guide covers what is specific to Kubernetes. Tasks that are the same for every G&L deployment, such as preparing the database, the post-install database steps and backups, are listed with a link to RSA's guide. AFX and remote agents run on separate servers and are not covered.

Values are hard to change later

Paths, the namespace and the image reference are written into the manifests and each node's Aveksa_System.cfg at Step 6, and G&L encrypts the passwords in that file on its first start. Confirm them, and the database time zone, before Step 6.

Architecture

Each G&L node runs as one pod. The kit gives each pod an external Service (LoadBalancer or NodePort) and an internal ClusterIP Service for the other G&L pods.

The schema is in Oracle, and the configuration and encryption keys are on NFS, so a pod can be recreated on another worker without data loss. Application logs are the exception: they are inside the pod (see Day-2 operations).

Component	Purpose	Location
`rsaiglmaster`	Master node; becomes the System Operations Node	Deployment in namespace `rsaigl`
`host1`, `host2`, ...	Host (general) nodes, cluster only	One Deployment each
`rsaiglsvcmaster`, `rsaiglsvchost1`	External Services: 8080, 8443, 8444	Services in `rsaigl`
`masterinternalsvc`, `internalsvc-host1`	Internal Services, including jgroups 7600 and 57600	Services in `rsaigl`
`iglconfig`	`Aveksa_System.cfg`, at `/home/oracle/config`	NFS, one directory per node
`iglkeystore`	WildFly password vault, at `/home/oracle/keystore`	NFS, shared
`iglsecurity`	Key encryption keys (KEK), at `/home/oracle/security`	NFS, shared, mode 700
AVDB	Schemas `AVUSER`, `AVDWUSER`, `AVCSUSER`, `ACMDB`, `PERFSTAT`	Customer database server

Network paths

From	To	Port	Purpose
Browsers	G&L external Service	8443	HTTPS to `/aveksa/main`
AFX, remote agents	G&L external Service	8444	Application communication
Every worker	Oracle	1555, or the customer's listener port	JDBC from the pod's worker
Every worker	NFS server	2049; also 111 and mountd for NFSv3	G&L volumes
Every node	Registry	The registry's port	Image pull
G&L pods	G&L pods	7600, 57600	jgroups discovery and failure detection (cluster)
Admin workstation	Registry, Kubernetes API	Registry port, 6443	Image push, `kubectl`

Port numbers for G&L are from the Installation Guide, Ch. 2, Port Requirements.

Prerequisites

Confirm every row before Step 1.

Item	Requirement	How to confirm
Kubernetes access	kubeconfig that can create namespaces, PersistentVolumes, Deployments and Services; `sudo` on a Linux host with `kubectl` (normally the control-plane node)	`kubectl auth can-i create persistentvolumes` returns `yes`
G&L workers	One worker per G&L node with at least 16 GB for the pod; WildFly heap 12288 MB (Installation Guide, Ch. 2)	Allocatable memory in `kubectl describe node <worker>`
Container runtime	containerd	`kubectl get nodes -o wide` shows `containerd://`
Registry	HTTPS endpoint, push and pull accounts, certificate trusted by every node	`curl -u <user> https://<registry>/v2/_catalog` from a worker
NFS	Export mounted at the same path on every worker that can run G&L	`mountpoint /var/nfs/general` on each worker
Service publishing	A LoadBalancer implementation and reserved addresses, or NodePort	Other LoadBalancer Services show an `EXTERNAL-IP`
Oracle 19c	Prepared for G&L as for any deployment, including the grants for P07 and later	The DBA, per the Database Setup and Management Guide and the P11 HF01 Release Notes
Database details	Host, port, SID, five schema passwords, database time zone	Provided by the DBA
Docker Hub or a mirror	The kit's host init container uses the image `nginx`	`crictl pull docker.io/library/nginx` on a worker

Media

File	Content
`ACM_Container_VP11_HF01_8.0.0.189610.tar`	G&L image at 8.0 P11 HF01 (about 4 GB); no separate patch step is needed
`ACM_Kubernetes_V8.0.0.zip`	`createYamlFiles.sh` and `README.md`

Workstation tools: kubectl, skopeo and curl.

Placeholders used in this guide

Placeholder	Meaning	Example
`<registry>`	Registry host and port	`registry.example.com:5000`
`<db-host>`	Oracle host	`oradb01.example.com`
`<nfs-server>`	NFS server	`nfs01.example.com`
`<tz>`	Database time zone (IANA name)	`America/New_York`
`<lb-ip>`, `<lb-ip-host1>`	Reserved Service addresses	`10.0.0.100`, `10.0.0.101`
`<worker-1>`, `<worker-2>`	Workers for the master and host1	`k8s-w1`, `k8s-w2`

Installation flow

Each step names where to run it and ends with a Check. Do not continue until the check passes.

Time required

Measured in a test deployment on a clean cluster and a new database: two workers with 4 vCPU and 9 GB each, a 6144 MB heap, and the registry on the same local network as the cluster. Other hardware and networks give different times. Steps not listed are manual.

Step	Measured	Notes
4. Push the image (3.9 GB tar, `skopeo`)	1.9 minutes	Network speed to the registry decides this
5. Pull test (both workers in parallel)	41 seconds	Leaves the image cached on each worker
6. `createYamlFiles.sh`	Seconds	Once the answers are ready
9. `run.sh`	1 second	Applies the manifests only
9. Master serving, from pod creation	4.4 minutes	WildFly started after 31 seconds; the first start's schema migration took 1.4 minutes (243 objects in `AVUSER`)
9. host1 serving, from pod creation	7.2 minutes	Its init container polls the master every 300 seconds; host1 joined the domain at 5.7 minutes
Redeploy: delete the namespace	11 to 48 seconds	Redeploy from scratch

Step 1: Confirm the Oracle database

Run on: each worker that can run G&L.

The database is prepared as for any G&L deployment, by RSA's database-only installer or by the DBA, following the Database Setup and Management Guide, Ch. 2, and the database grants in the P11 HF01 Release Notes. G&L creates its objects in AVUSER on its first start.

Collect from the DBA:

Value	Used in
Database host	Step 6, prompt 3
Listener port and SID (G&L defaults: 1555, AVDB)	Step 7
Passwords of `AVUSER`, `AVDWUSER`, `AVCSUSER`, `ACMDB`, `PERFSTAT`	Steps 6 and 7
Database time zone, as an IANA name	Step 8

G&L connects to the database from its pod, so from whichever worker runs the pod, not from the control-plane node.

Check: each worker reaches the listener:

timeout 5 bash -c 'exec 3<>/dev/tcp/<db-host>/1555' && echo reachable
# reachable

Step 2: Stage the kit

Run on: the control-plane node, as a user with sudo. The kit's README requires the script to run on the master node.

mkdir -p ~/gl-k8s && cd ~/gl-k8s
unzip /path/to/ACM_Kubernetes_V8.0.0.zip
chmod 755 createYamlFiles.sh

Check: createYamlFiles.sh is executable, and sudo kubectl get nodes shows every node Ready. The generated run.sh uses sudo kubectl, so root needs a working kubeconfig.

Step 3: Prepare the shared volumes

Run on: the NFS server, then every worker, as root.

Why every worker needs the mount

The kit does not use a StorageClass. It writes hostPath PersistentVolumes that point at directories on the node, and claims that bind to them:

A hostPath volume is the directory on whichever worker runs the pod. Without the NFS mount, the pod gets an empty local directory and cannot find its configuration or keys.

NFS server

The directory names are RSA's examples (Installation Guide, Ch. 2, Kubernetes Prerequisites).

dnf install -y nfs-utils                       # Ubuntu: apt install nfs-kernel-server
mkdir -p /var/nfs/general
echo '/var/nfs/general <worker-1-ip>(rw,sync,no_subtree_check) <worker-2-ip>(rw,sync,no_subtree_check)' >> /etc/exports
systemctl enable --now nfs-server
exportfs -ra

cd /var/nfs/general
mkdir dataiglconfig76 dataiglkeystore76 dataiglsecurity76
mkdir dataiglconfig761                         # cluster: one config directory per host
chown 1000:1000 dataiglconfig76 dataiglkeystore76 dataiglsecurity76 dataiglconfig761
chmod 755 dataiglconfig76 dataiglkeystore76 dataiglconfig761
chmod 700 dataiglsecurity76

Owner is uid 1000

RSA's prerequisites show chown oracle:oninstall. The 8.0 P11 HF01 image has no oracle user; it runs as jboss, uid 1000, gid 1000. Step 5 confirms this from the image.

In a cluster, each host has its own config directory and shares the master's keystore and security directories.

Every worker, and the control-plane node

createYamlFiles.sh (Step 6) writes each node's Aveksa_System.cfg into the config directory on the node where it runs, so that node needs the same mount.

dnf install -y nfs-utils                       # Ubuntu: apt install nfs-common
mkdir -p /var/nfs/general
echo '<nfs-server>:/var/nfs/general /var/nfs/general nfs defaults,_netdev 0 0' >> /etc/fstab
mount /var/nfs/general

Check: on every worker and the control-plane node, the share is mounted, owned by uid 1000, and the security directory is mode 700:

mountpoint /var/nfs/general
# /var/nfs/general is a mountpoint

ls -ldn /var/nfs/general/dataigl*
# drwxr-xr-x. 2 1000 1000 31 Oct  6 03:13 /var/nfs/general/dataiglconfig76
# drwxr-xr-x. 2 1000 1000 31 Oct  5 23:07 /var/nfs/general/dataiglconfig761
# drwxr-xr-x. 2 1000 1000 91 Oct  6 09:36 /var/nfs/general/dataiglkeystore76
# drwx------. 2 1000 1000 36 Oct  5 22:52 /var/nfs/general/dataiglsecurity76

Step 4: Push the G&L image

Run on: the admin workstation.

RSA's Kubernetes chapter loads the image with docker load on each node. Showstopper on current clusters: Kubernetes 1.24 removed dockershim, and containerd does not see images in Docker's store (Dockershim removal FAQ). Workaround: push the image to the registry, which the kit's README also allows ("one can have a private registry with the image uploaded in it"):

Read the tag recorded in the tar, then copy the tar to the registry with skopeo, which needs no container engine:

tar -xOf ACM_Container_VP11_HF01_8.0.0.189610.tar manifest.json      # RepoTags: identity/rsaigl:8.0.0.189610

skopeo copy --dest-creds '<push-user>:<password>' --dest-cert-dir <dir-with-ca.crt> \
  docker-archive:ACM_Container_VP11_HF01_8.0.0.189610.tar \
  docker://<registry>/rsa/acm:8.0.0.189610

--dest-cert-dir is needed only for a private CA. That CA must also be in every node's operating system trust store (for example /etc/pki/ca-trust/source/anchors/, then update-ca-trust), or pulls fail with x509: certificate signed by unknown authority.

Check:

curl -u <user> https://<registry>/v2/rsa/acm/tags/list       # {"name":"rsa/acm","tags":["8.0.0.189610"]}

Step 5: Create the namespace and pull secret

Run on: the control-plane node.

The kit's manifests reference no pull secret. Create one in the namespace (Pull an image from a private registry):

kubectl create namespace rsaigl
kubectl create secret docker-registry gl-registry -n rsaigl \
  --docker-server=<registry> --docker-username=<pull-user> --docker-password='<password>'

Test the pull on each G&L worker. The pod also prints the user the image runs as, and leaves the 4 GB image cached on the worker:

apiVersion: v1
kind: Pod
metadata:
  name: gl-pull-test
  namespace: rsaigl
spec:
  nodeName: <worker-1>
  restartPolicy: Never
  imagePullSecrets:
  - name: gl-registry
  containers:
  - name: probe
    image: <registry>/rsa/acm:8.0.0.189610
    imagePullPolicy: IfNotPresent
    command: ["sh", "-c", "id"]

kubectl apply -f gl-pull-test.yaml
kubectl logs gl-pull-test -n rsaigl          # uid=1000(jboss) gid=1000(jboss) groups=1000(jboss)
kubectl delete pod gl-pull-test -n rsaigl

Repeat with nodeName: <worker-2> in a cluster.

Check: the pod completes on every G&L worker and prints uid 1000. If the uid differs, use it for the chown commands in Steps 3 and 7.

Step 6: Generate the manifests

Run on: the control-plane node, in ~/gl-k8s.

createYamlFiles.sh writes the manifests and each node's Aveksa_System.cfg, including the cluster parameters, into the config paths it is given on this node. It writes the file only when the config directory has none; an existing file is used as it is (Installation Guide, Ch. 4). Leave the config directories empty, or the nodes will not form a cluster. The directories belong to uid 1000 (Step 3), so run the script with sudo.

ls /var/nfs/general/dataiglconfig76 /var/nfs/general/dataiglconfig761    # no Aveksa_System.cfg
sudo ./createYamlFiles.sh

Prompts in the order the 8.0 kit asks them. Enter accepts the default in brackets.

#	Prompt	Answer
1	How many IGL application host's nodes	`0` single node, `1` master and host1
2	Provide the image name	`<registry>/rsa/acm:8.0.0.189610`
3	Remote Database IP/Hostname	`<db-host>`
4	AVUSER password, then confirmation	The `AVUSER` password
5	Master host name `[rsaiglmaster]`	Enter
6	Master service name `[rsaiglsvcmaster]`	Enter
7	Master's config path	`/var/nfs/general/dataiglconfig76`
8	Master's keystore path	`/var/nfs/general/dataiglkeystore76`
9	Master's security path	`/var/nfs/general/dataiglsecurity76`
10	Namespace `[rsaigl]`	Enter
11	LoadBalancer Hostname/IP	`<lb-ip>`
12	Port used to access the application	`8443`
13	Service Type `[NodePort/LoadBalancer]`	`LoadBalancer`, or `NodePort` without a load balancer
14	host1 hostname `[host1]` (cluster)	Enter
15	host1 config path (cluster)	`/var/nfs/general/dataiglconfig761`
16	LoadBalancer Hostname/IP (cluster)	`<lb-ip-host1>`
17	Port used to access the application (cluster)	`8443`

The Installation Guide's prompt list omits prompts 11, 12, 16 and 17.

The full session for a cluster of the master and host1

By default, this would create manifest for domain node. How many IGL application host's nodes are you planning to deploy in the Kubernetes Cluster : 1

Provide the image name : <registry>/rsa/acm:8.0.0.189610

Enter the Remote Database IP/Hostname : <db-host>
Provided remote database is :  <db-host>

Enter the AVUSER password :
Confirm the AVUSER password :

Enter the IGL application master host name [rsaiglmaster]:
The provided IGL application master hostname is :  rsaiglmaster

Enter the IGL application master service name [rsaiglsvcmaster]:
The entered IGL application master service name is :  rsaiglsvcmaster

Enter master's config path : /var/nfs/general/dataiglconfig76
Provided master's config path is :  /var/nfs/general/dataiglconfig76

Enter master's keystore path : /var/nfs/general/dataiglkeystore76
Provided master's keystore path is :  /var/nfs/general/dataiglkeystore76

Enter master's security path : /var/nfs/general/dataiglsecurity76
The provided master's security path is :  /var/nfs/general/dataiglsecurity76

Enter the namespace in which the rsaigl components to be deployed in the cluster [rsaigl]:
The provided namespace is :  rsaigl

Enter the LoadBalancer Hostname/IP required to access the application (Leave it empty if the value is unknown and edit it later) : <lb-ip>

Enter the port used to access the application (Leave it empty if the value is unknown and edit it later) : 8443

Enter the service Type for the Pods [NodePort/LoadBalancer] : LoadBalancer
Provided service type is :  LoadBalancer

### Creating the Aveksa_System.cfg file in the master config directory ###
----------------------------------------------------------------
The entered Pod Service type is :  LoadBalancer
----------------------------------------------------------------
Creating run.sh file
----------------------------------------------------------------
Number of IGL application hosts you selected to deploy in the K8s cluster is : 1

Enter the IGL application host1 hostname [host1]:
Provided host1 hostname is : host1

Enter the host1 config path : /var/nfs/general/dataiglconfig761
Provided host1 config path is :  /var/nfs/general/dataiglconfig761

Enter the LoadBalancer Hostname/IP required to access the application (Leave it empty if the value is unknown and edit it later) : <lb-ip-host1>

Enter the port used to access the application (Leave it empty if the value is unknown and edit it later) : 8443

### Creating Aveksa_System.cfg file under the given config path ie., /var/nfs/general/dataiglconfig761  ###

### Creating of Aveksa_System.cfg file for host1 is completed  ###

The cluster parameters the kit writes:

Key	Master (`dataiglconfig76`)	host1 (`dataiglconfig761`)
`NODE_TYPE`	`Domain`	`Host`
`HOST_XML_NAME`	`master`	`slave1`
`SERVER_NAME`	`img-server-master`	`img-server-1`
`CLUSTER_NODES_IP`	`rsaiglmaster,host1`
`DOMAIN_MASTER`		`masterinternalsvc`
`DOMAIN_USER`, `DOMAIN_PASSWORD`, `SERVER_GROUP`	Kit defaults	Same as the master

Generated files:

File	Content
`namespace.yaml`	Namespace `rsaigl`
`master.yaml`	Three PersistentVolumes, three claims, the master Deployment
`service_master.yaml`	External Service: 8080, 8443, 8444
`service_internal_master.yaml`	Internal Service, including 9999, 7600 and 57600
`host1.yaml`, `service_host1.yaml`, `service_internal_host1.yaml`	The same for host1 (cluster)
`run.sh`	`kubectl apply` for each file, in order

Check: the script prints "### Creating the Aveksa_System.cfg file in the master config directory ###" (and, in a cluster, the same for host1), and each node's file has its role:

sudo grep -H NODE_TYPE /var/nfs/general/dataiglconfig76*/Aveksa_System.cfg
# /var/nfs/general/dataiglconfig761/Aveksa_System.cfg:NODE_TYPE=Host
# /var/nfs/general/dataiglconfig76/Aveksa_System.cfg:NODE_TYPE=Domain

Step 7: Complete Aveksa_System.cfg

Run on: the NFS server, as root.

The kit's file holds the database host, the AVUSER password, the load balancer address and the cluster parameters. Add the keys it has no prompt for to each node's file (Installation Guide, Ch. 2, Verify Docker Container Requirements):

for d in /var/nfs/general/dataiglconfig76 /var/nfs/general/dataiglconfig761; do
cat >> $d/Aveksa_System.cfg <<'EOF'
REMOTE_ORACLE_PORT=1555
REMOTE_ORACLE_DATABASE=AVDB
AVEKSA_USER=avuser
AVEKSA_REPORTS_USER=avdwuser
AVEKSA_REPORTS_PASS=<avdwuser-password>
AVEKSA_CSUSER=avcsuser
AVEKSA_CSUSER_PASS=<avcsuser-password>
AVEKSA_PUBLIC_DB_USER=ACMDB
AVEKSA_PUBLIC_DB_PASS=<acmdb-password>
AVEKSA_AVPERF_USER=perfstat
AVEKSA_AVPERF_PASS=<perfstat-password>
WILDFLY_HEAP_MEM=12288
AMAZON_RDS=N
EOF
chown 1000:1000 $d/Aveksa_System.cfg
chmod 777 $d/Aveksa_System.cfg
done

Showstopper: host password

The kit writes every host's file with AVEKSA_PASS=secret, whatever AVUSER password was entered. Workaround: set it to the AVUSER password in each host's file:

sed -i 's/^AVEKSA_PASS=.*/AVEKSA_PASS=<avuser-password>/' /var/nfs/general/dataiglconfig761/Aveksa_System.cfg

Key	Notes
`REMOTE_ORACLE_PORT`, `REMOTE_ORACLE_DATABASE`	Defaults 1555 and AVDB
`AVEKSA_CSUSER`	One table in the Installation Guide spells it `AVESKA_CSUSER`; the sample file and the product use `AVEKSA_CSUSER`
`WILDFLY_HEAP_MEM`	Heap in MB; RSA's minimum is 12288

Plain text only until the first start

G&L encrypts the passwords in this file on its first start and writes it back, which is why it must be writable by uid 1000.

Check: in each config directory, ls -ln Aveksa_System.cfg shows 1000 1000 and -rwxrwxrwx, cut -d= -f1 Aveksa_System.cfg lists 24 keys, and NODE_TYPE is still present.

Step 8: Correct the manifests

Run on: the control-plane node, in ~/gl-k8s.

The kit's output does not deploy a working G&L on Kubernetes as generated. The Installation Guide allows the generated YAML to be edited (Ch. 4). The changes are grouped into kit defects that stop the deployment, an RSA requirement the kit has no prompt for, and recommendations. Keep the originals first:

for f in *.yaml; do cp -n "$f" "$f.orig"; done

#	Kit output	Result without the change	Type
1	Claims name no volume	Claims bind to the wrong volumes; the master exits with code 254 and writes no log	Showstopper
2	No `imagePullSecrets`	`ErrImagePull: no basic auth credentials` from a registry that requires a login	Showstopper
3	Host names (`rsaiglmaster`, `host1`) not resolvable between pods	The nodes cannot discover each other	Showstopper (cluster)
4	Cluster member list applied after WildFly starts	Every server runs with `127.0.0.1[7600]`; no node finds another	Showstopper (cluster)
5	No `TZ`	Container time zone differs from the database's, which RSA does not allow	RSA requirement
6	`imagePullPolicy` not set	The 4 GB image may be pulled again on restarts	Recommended
7	`15Gi` memory request, no placement	Pod `Pending` if no worker fits; placement left to the scheduler	Recommended
8	No fixed LoadBalancer address	The Service address can change when it is recreated	Recommended

Showstopper 1: bind each claim to its volume

All kit volumes are identical apart from their path (storageClassName: manual, 8Gi, ReadWriteMany), and the claims name no volume, so Kubernetes binds a claim to any free volume. In a cluster deployment, configmaster-claim was bound to a keystore volume and securitymaster-claim to a config volume. Workaround: reserve each volume with volumeName (Reserving a PersistentVolume):

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: configmaster-claim
spec:
  storageClassName: manual
  volumeName: configmaster
  accessModes:
    - ReadWriteMany

Claim	`volumeName`
`configmaster-claim`, `securitymaster-claim`, `keystoremaster-claim`	`configmaster`, `securitymaster`, `keystoremaster`
`confighost1-claim`, `securityhost1-claim`, `keystorehost1-claim`	`config1`, `security1`, `keystore1`

Showstopper 2: pull secret

Workaround: add the secret from Step 5 to each Deployment, under spec.template.spec:

      imagePullSecrets:
      - name: gl-registry

Showstopper 3 (cluster): host names

CLUSTER_NODES_IP lists the nodes by host name, and each node must resolve the others' names to their pod addresses: the Installation Guide's fix copies each container's address into every container's /etc/hosts (Ch. 4, Common Problem). Kubernetes rewrites /etc/hosts at each pod start and gives a recreated pod a new address. A name that resolves to a Service's ClusterIP is not enough: discovery through it finds no members. Workaround: a headless Service per host name, which resolves the name to the pod's own address in cluster DNS (Headless Services). Add one to each internal Service file:

# appended to service_internal_master.yaml; service_internal_host1.yaml gets
# the same with name: host1 and selector name: host1
---
apiVersion: v1
kind: Service
metadata:
  name: rsaiglmaster
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    name: rsaiglmaster
  ports:
  - name: jgroups-tcp
    port: 7600
  - name: jgroups-tcp-fd
    port: 57600

Showstopper 4 (cluster): member list before start

The image's start script, /home/oracle/execute.sh, writes CLUSTER_NODES_IP into WildFly's domain.xml only after WildFly has started, and does not reload it. Every Kubernetes pod starts from the image, so the servers always run with the image's 127.0.0.1[7600]. Workaround: set the same value before execute.sh runs, in the master's G&L container. The master is the domain controller and passes the setting to every host.

# master.yaml, G&L container
        command: ["/bin/sh", "-c"]
        args:
        - |
          f=/home/oracle/config/${CONFIG_FILE:-Aveksa_System.cfg}
          v=$(sed -n "s/^CLUSTER_NODES_IP=//p" "$f" | sed -E "s/([^,]+)/\1[7600]/g")
          [ -n "$v" ] && sed -i -E "s|(name=\"jboss.cluster.tcp.initial_hosts\"[[:space:]]+value=\")[^\"]*\"|\1$v\"|" /opt/jboss/wildfly/domain/configuration/domain.xml
          exec /home/oracle/execute.sh

The edit replaces only the value between the quotes. The image writes the element as value="127.0.0.1[7600]" />, with a space before />, so a pattern that expects "/> matches nothing and the servers still start with 127.0.0.1[7600].

RSA requirement 5: time zone

The container must use the database's time zone; RSA's Docker procedure passes -e TZ when the database is not on America/New_York (Installation Guide, Ch. 5). The kit has no prompt for it. Add it to each G&L container:

        env:
        - name: TZ
          value: <tz>

Recommended 6 to 8

      nodeSelector:
        kubernetes.io/hostname: <worker-1>      # 7: <worker-2> in host1.yaml
      containers:
      - name: rsaiglmaster
        imagePullPolicy: IfNotPresent          # 6

For 7, compare the 15Gi request with each worker's allocatable memory (kubectl get nodes -o custom-columns=NODE:.metadata.name,ALLOCATABLE:.status.allocatable.memory). A pod whose request exceeds every worker stays Pending (Resource management). Size the workers rather than lowering the request: a worker without enough memory kills WildFly as the heap grows (OOMKilled, exit code 137), possibly after the pod reported Ready.

For 8, the method depends on the load balancer. For MetalLB (MetalLB usage):

metadata:
  name: rsaiglsvcmaster
  annotations:
    metallb.universe.tf/loadBalancerIPs: <lb-ip>

Check: for f in *.yaml; do diff "$f.orig" "$f"; done shows only the changes above. For master.yaml in a cluster, the additions are:

>   volumeName: configmaster
>   volumeName: securitymaster
>   volumeName: keystoremaster
>         - name: TZ
>           value: <tz>
>         imagePullPolicy: IfNotPresent
>         command: ["/bin/sh", "-c"]
>         args:
>         - |
>           f=/home/oracle/config/${CONFIG_FILE:-Aveksa_System.cfg}
>           ...
>           exec /home/oracle/execute.sh
>       imagePullSecrets:
>       - name: gl-registry
>       nodeSelector:
>         kubernetes.io/hostname: <worker-1>

host1.yaml has the same changes without command and args, and each service_internal_*.yaml gains its headless Service.

Step 9: Deploy

Run on: the control-plane node, in ~/gl-k8s.

cat run.sh
./run.sh
kubectl get pods -n rsaigl -o wide -w

The first start creates the schema and is the slowest. Follow it with kubectl logs -f deploy/rsaiglmaster-deployment -n rsaigl.
In a cluster, host1 stays Init:0/1 until the master answers on http://masterinternalsvc:8080; the init container checks every 300 seconds.
The kit defines no readiness probe for the application, so Running does not mean G&L is serving. Step 10 confirms it.

A cluster deployment, watched with timestamps. host1 waits in its init container until the master serves, then starts:

TIME      EVENT      NAME                                       READY   STATUS            AGE     NODE
36:05  ADDED      host1-deployment-6599cf7489-n896j          0/1     Init:0/1          2m30s   <worker-2>
36:05  ADDED      rsaiglmaster-deployment-5bff7797df-tdm8d   1/1     Running           2m30s   <worker-1>
39:07  MODIFIED   host1-deployment-6599cf7489-n896j          0/1     PodInitializing   5m32s   <worker-2>
39:07  MODIFIED   host1-deployment-6599cf7489-n896j          1/1     Running           5m32s   <worker-2>

The master's container log shows the start sequence. The lines to look for:

Creating User for Domain Management
Created Domain Management user -  AveksaClusterAdmin
Fetching latest keystore password from the Database
Successfully created New Password Vault Keystore for wildfly server
Password Vault configured successfully...
Starting Domain-Controller Wildfly...
Wildfly Started Successfully
Deploying aveksa.ear...
aveksa.ear Deployed Successfully
Deploying aveksaWFArchitect.ear...
aveksaWFArchitect.ear Deployed Successfully
Updating passed DB credentials with encrypted values

Check: every pod is Running and 1/1, and each external Service has an EXTERNAL-IP:

kubectl get pods,svc -n rsaigl -o wide

NAME                                           READY   STATUS    RESTARTS   AGE     NODE
pod/host1-deployment-6599cf7489-n896j          1/1     Running   0          9m37s   <worker-2>
pod/rsaiglmaster-deployment-5bff7797df-tdm8d   1/1     Running   0          9m37s   <worker-1>

NAME                        TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)
service/host1               ClusterIP      None            <none>          7600/TCP,57600/TCP
service/internalsvc-host1   ClusterIP      10.99.67.190    <none>          1158/TCP,1155/TCP,...,9999/TCP,7600/TCP,57600/TCP
service/masterinternalsvc   ClusterIP      10.101.73.9     <none>          1158/TCP,1155/TCP,...,9999/TCP,7600/TCP,57600/TCP
service/rsaiglmaster        ClusterIP      None            <none>          7600/TCP,57600/TCP
service/rsaiglsvchost1      LoadBalancer   10.99.202.189   <lb-ip-host1>   8080:30351/TCP,8443:32738/TCP,8444:32726/TCP
service/rsaiglsvcmaster     LoadBalancer   10.96.158.145   <lb-ip>         8080:30401/TCP,8443:30872/TCP,8444:31124/TCP

rsaiglmaster and host1 are the headless Services added in Step 8.

Step 10: Verify the deployment

Run on: the control-plane node, then a browser. Do not hand over until every check passes.

1. Pods are steady. RESTARTS is 0 on every pod.

kubectl get pods -n rsaigl
# NAME                                       READY   STATUS    RESTARTS   AGE
# host1-deployment-6599cf7489-n896j          1/1     Running   0          7m48s
# rsaiglmaster-deployment-5bff7797df-tdm8d   1/1     Running   0          7m48s

2. Every node answers on its own address. Each external Service returns 200 or 30x. A host starts after the master (its init check runs every 300 seconds), so retest a host's 000 before treating it as a failure.

curl -k -s -o /dev/null -w '%{http_code}\n' https://<lb-ip>:8443/aveksa/main          # 200
curl -k -s -o /dev/null -w '%{http_code}\n' https://<lb-ip-host1>:8443/aveksa/main    # 200

3. WildFly started and the application is deployed. The container log has the start lines and none of WFLYSRV0026, WFLYCTL0186 or rolled back. Ignore WFLYSRV0003: Could not index class warnings; they are normal.

for d in rsaiglmaster-deployment "host1-deployment -c host1"; do
  kubectl logs deploy/$d -n rsaigl | grep -E 'Starting (Domain|Host)-Controller|Wildfly Started Successfully|aveksa.ear Deployed Successfully|WFLYSRV0026|WFLYCTL0186|rolled back'
done
# Starting Domain-Controller Wildfly...
# Wildfly Started Successfully
# aveksa.ear Deployed Successfully
# Starting Host-Controller Wildfly...
# Wildfly Started Successfully

A host does not deploy aveksa.ear itself: the master, as domain controller, deploys it to every server in the domain.

4. Version. G&L's logs are inside the pod, under /opt/jboss/wildfly/domain/servers/<SERVER_NAME>/log (img-server-master on the master, img-server-1 on host1).

POD=$(kubectl get pod -n rsaigl -l name=rsaiglmaster -o name)
LOG=$(kubectl exec -n rsaigl $POD -- sh -c 'ls -d /opt/jboss/wildfly/domain/servers/*/log')
kubectl exec -n rsaigl $POD -- grep -m1 '^Version' $LOG/aveksaServerInfo.log
# Version 8.0.0.189610 P11_HF01

5. First logon. Open https://<lb-ip>:8443/aveksa/main and sign in as AveksaAdmin with aveksa123. G&L asks for a new password at once: at least eight characters, with upper case, lower case and a number (Installation Guide, Ch. 3).

6. Server Nodes. Go to Admin > System > Server Nodes. Each G&L node registers itself here when it starts, named <host name>-<SERVER_NAME>. Expect one row per node: exactly one System Operations Node, the rest General Node, all Healthy, with a recent heartbeat. A general node designated with Make Next SON shows (Next SON) after its role; see System Operations Node handover.

Admin > System > Server Nodes in a two-node cluster: rsaiglmaster-img-server-master as the System Operations Node and host1-img-server-1 as a General Node marked Next SON, both Healthy

A node that was renamed or removed keeps its row, with its heartbeat stopped; delete it once it shows Shutdown or Unresponsive.

7. The nodes form one cluster (cluster only).

Server Nodes is not proof

Every node registers itself in the database, so Server Nodes lists all of them as Healthy even when each server runs alone. jms_status in t_server_nodes also reads HEALTHY in that state, so it is no proof either. Both checks below must pass.

a. host1 joined the master's domain. host1 starts as a host controller and connects to the master; the master registers it.

kubectl logs deploy/host1-deployment -c host1 -n rsaigl | grep -E 'Starting (Domain|Host)-Controller'
# Starting Host-Controller Wildfly...

H=$(kubectl get pod -n rsaigl -l name=host1 -o name)
kubectl exec -n rsaigl $H -c host1 -- grep 'Connected to master host controller' /opt/jboss/wildfly/domain/log/host-controller.log
# WFLYHC0148: Connected to master host controller at remote://masterinternalsvc:9999

M=$(kubectl get pod -n rsaigl -l name=rsaiglmaster -o name)
kubectl exec -n rsaigl $M -- grep 'Registered remote slave host' /opt/jboss/wildfly/domain/log/host-controller.log
# WFLYHC0019: Registered remote slave host "slave1", ...

Starting Domain-Controller Wildfly on host1 means it is running as its own domain: see Step 6.

b. One cluster view with every node in it. Each server's latest view counts all the nodes, here (2) with both names:

for p in $M "$H -c host1"; do
  kubectl exec -n rsaigl $p -- sh -c 'grep ISPN000094 /opt/jboss/wildfly/domain/servers/*/log/server.log | tail -1'
done
# ISPN000094: Received new cluster view for channel ejb: [master:img-server-master|1] (2) [master:img-server-master, slave1:img-server-1]

A view of (1), or no members discovered ... creating cluster as coordinator, means the servers did not find each other: see Step 8, changes 3 and 4. The member list a server started with is in its JVM arguments:

kubectl exec -n rsaigl $M -- sh -c 'for d in /proc/[0-9]*; do tr "\0" "\n" < $d/cmdline 2>/dev/null | grep "^-Djboss.cluster.tcp.initial_hosts"; done'
# -Djboss.cluster.tcp.initial_hosts=rsaiglmaster[7600],host1[7600]

8. Pods are still steady. Repeat check 1. A worker short of memory can kill WildFly minutes after it reported Ready.

Step 11: Post-install tasks and backup

Run on: the database server (the DBA), then the control-plane node and the NFS server.

Database tasks. These are the same for every G&L deployment and need objects that G&L creates on its first start: recompile AVUSER.Utilities_Pkg, confirm the additional AVUSER grants and the sys_perfstat migration row, and compare the database time zone with G&L's, as AVUSER (Database Setup and Management Guide, Ch. 2, Additional AVUSER Schema Permissions). If the time zones differ, the DBA corrects the database's; then restart the G&L pods (Stop and start).

Backup. Back up the database and the key encryption keys together; without the keys, encrypted data in the database cannot be recovered (Database Setup and Management Guide, Ch. 3). On Kubernetes the keys are in the security directory on NFS, and G&L is stopped and started with kubectl:

kubectl scale deployment --all --replicas=0 -n rsaigl                          # control-plane node
# database export by the DBA, per the Database Setup and Management Guide, Ch. 3
tar -C /var/nfs/general -czf /root/gl-kek-$(date +%F).tgz dataiglsecurity76  # NFS server
kubectl scale deployment rsaiglmaster-deployment --replicas=1 -n rsaigl        # then the hosts

Check: the DBA confirms the database tasks, and the database export and the key archive carry the same date and are stored together outside the cluster.

Day-2 operations

Stop and start

kubectl scale deployment --all --replicas=0 -n rsaigl
kubectl scale deployment rsaiglmaster-deployment --replicas=1 -n rsaigl
kubectl scale deployment host1-deployment --replicas=1 -n rsaigl     # cluster, after the master

Every start creates new containers, so confirm the cluster again with Step 10, check 7.

Logs

The kit persists only config, keystore and security. WildFly and G&L logs are lost when a pod is deleted. Copy them first:

kubectl cp rsaigl/<pod>:/opt/jboss/wildfly/domain/servers/<SERVER_NAME>/log ./gl-logs-<pod>

Redeploy from scratch

Deleting the namespace does not delete the PersistentVolumes, which are cluster-scoped, and it does delete the pull secret. Before running run.sh again:

kubectl delete namespace rsaigl
kubectl get pv                                    # the kit's volumes show Released
kubectl delete pv configmaster securitymaster keystoremaster config1 security1 keystore1

Then repeat Step 5 (namespace and pull secret) and Step 9. The data on NFS, including each Aveksa_System.cfg, is not affected: the volumes are hostPath with the Retain policy.

Newer image

A container is not patched in place. Back up (Step 11) and push the new image (Step 4). In a cluster, scale the hosts to zero, change the master's image: and apply it, let the master migrate the schema alone, then start the hosts on the same image. A clustered deployment that is migrated or patched must run the System Operations Node without any other active node (Installation Guide, Ch. 7; Patch Installation Guide).

System Operations Node handover

The first node started becomes the System Operations Node (SON). G&L does not fail over automatically: another node becomes SON only when an administrator designates it (Administrator's Guide, Ch. 2, Managing Server Cluster Nodes).

Plan a successor: in Admin > System > Server Nodes, click Make Next SON on an active general node. It becomes SON the next time it starts after the current SON has been shut down.
Replace a SON that is down: with the current SON shut down, click Make SON on the node to promote.
Server Nodes can modify only inactive nodes (Shutdown or Unresponsive); use Delete to remove a node that is no longer part of the cluster.

RSA recommends directing user traffic to general nodes only. SON tasks such as collections may depend on files on the SON, so test them after a handover.

Troubleshooting

Start with:

kubectl get events -n rsaigl --sort-by=.lastTimestamp | tail -20
kubectl describe pod <pod> -n rsaigl
kubectl logs <pod> -n rsaigl --previous

Pod Does Not Start

Symptom: ErrImagePull: no basic auth credentials

Root Cause: Image pull secret is missing in the namespace (for example, after recreating the namespace).
Resolution: Re-create the pull secret as documented in Step 5.

Symptom: ErrImagePull: x509: certificate signed by unknown authority

Root Cause: Private registry CA certificate is not trusted by the Kubernetes worker node.
Resolution: Install and trust the registry CA certificate on each worker node per Step 4.

Symptom: ErrImagePull: not found

Root Cause: Incorrect image registry reference, repository path, or tag.
Resolution: Verify image name and tag in Step 4 and prompt 2 of Step 6.

Symptom: Pending: Insufficient memory

Root Cause: The 15Gi memory request cannot be scheduled on any single worker node.
Resolution: Scale worker node memory or adjust memory requests per Step 8, change 7.

Symptom: Pending: unbound immediate PersistentVolumeClaims; volumes Released

Root Cause: Orphaned PersistentVolumes remain from a prior deployment.
Resolution: Execute cleanup steps in Redeploy from scratch.

Symptom: Pod host1 remains in Init:0/1

Root Cause: Master node has not answered within the probe window, or the nginx container image cannot be pulled.
Resolution: Allow up to 300 seconds after master responds, or mirror the nginx image locally.

Pod Starts, Then Fails

Symptom: Exit code 254 (no log produced)

Root Cause: PersistentVolumeClaims are bound to the wrong volumes.
Resolution: Verify volume bindings using kubectl get pvc -n rsaigl and apply Step 8, change 1.

Symptom: OOMKilled (exit code 137)

Root Cause: Container memory consumption exceeded the node memory ceiling.
Resolution: Increase worker memory or tune resource limits per Step 8, change 7.

Symptom: Permission denied on /home/oracle/security

Root Cause: NFS directory is not owned by UID 1000 or file permissions are not 0700.
Resolution: Correct directory permissions and ownership as shown in Step 3.

Symptom: Empty configuration or regenerated encryption keys

Root Cause: Worker node is missing the shared NFS volume mount.
Resolution: Confirm the NFS export is mounted across all worker nodes per Step 3.

Symptom: Database connection errors in log

Root Cause: Incorrect database credentials in Aveksa_System.cfg (such as the default AVEKSA_PASS=secret), or the Oracle listener port is blocked from the worker node.
Resolution: Validate database connectivity and credentials in Step 1 and Step 7.

G&L Runs But Has Irregularities

Symptom: Node URL returns HTTP 000

Root Cause: Target pod is not ready or is restarting.
Resolution: Run kubectl describe pod <pod> -n rsaigl to verify pod health and probe status.

Symptom: Server Nodes UI lists fewer nodes than active pods

Root Cause: Host pod is not registered with the domain master controller (host log lacks Connected to master host controller).
Resolution: Confirm DOMAIN_MASTER in Step 6 and verify port 9999 connectivity on masterinternalsvc.

Symptom: Container log displays Starting Domain-Controller Wildfly on host

Root Cause: Aveksa_System.cfg was not created by createYamlFiles.sh (missing NODE_TYPE=Host), running the host as an independent domain.
Resolution: Regenerate manifests with NODE_TYPE=Host in Step 6.

Symptom: All nodes Healthy, but cluster view displays (1) member; server.log reports no members discovered

Root Cause: Server started with -Djboss.cluster.tcp.initial_hosts=127.0.0.1[7600], or host names resolve to a Service IP instead of pod IP.
Resolution: Apply Step 8, changes 3 and 4.

Symptom: EXTERNAL-IP remains <pending>

Root Cause: Cluster lacks a load balancer controller or has exhausted external IP pool.
Resolution: Provision a cluster load balancer (such as MetalLB) or change the Service type to NodePort.

Symptom: Timestamps are shifted

Root Cause: Container timezone differs from the Oracle Database timezone.
Resolution: Align container and database timezone configurations per Step 8, change 5, and Step 11.

Symptom: AFX cannot connect

Root Cause: Port 8444 is unreachable on the external Service.
Resolution: Open firewall port 8444 from AFX to the external Service IP.

References

RSA Governance & Lifecycle 8.0

RSA Governance & Lifecycle documentation (all guides and release notes, including 8.0 P11 HF01)
Installation Guide 8.0: Ch. 2 requirements and ports, Ch. 4 Kubernetes, Ch. 5 Docker cluster parameters, Ch. 7 patching clusters
Database Setup and Management Guide 8.0: Ch. 2 database configuration, Ch. 3 backup
Upgrade and Migration Guide 8.0
Patch Installation Guide 8.0: one node during a patch
8.0 Patch 11 HF01 Release Notes: database grants for P07 and later
Administrator's Guide 8.0: Ch. 2, Managing Server Cluster Nodes

Kubernetes

Registry and load balancing

Overview​

Architecture​

Network paths​

Prerequisites​

Installation flow​

Time required​

Step 1: Confirm the Oracle database​

Step 2: Stage the kit​

Step 3: Prepare the shared volumes​

Why every worker needs the mount​

NFS server​

Every worker, and the control-plane node​

Step 4: Push the G&L image​

Step 5: Create the namespace and pull secret​

Step 6: Generate the manifests​

Step 7: Complete Aveksa_System.cfg​

Step 8: Correct the manifests​

Showstopper 1: bind each claim to its volume​

Showstopper 2: pull secret​

Showstopper 3 (cluster): host names​

Showstopper 4 (cluster): member list before start​

RSA requirement 5: time zone​

Recommended 6 to 8​

Step 9: Deploy​

Step 10: Verify the deployment​

Step 11: Post-install tasks and backup​

Day-2 operations​

Stop and start​

Logs​

Redeploy from scratch​

Newer image​

System Operations Node handover​

Troubleshooting​

Pod Does Not Start​

Pod Starts, Then Fails​

G&L Runs But Has Irregularities​

References​

Overview

Architecture

Network paths

Prerequisites

Installation flow

Time required

Step 1: Confirm the Oracle database

Step 2: Stage the kit

Step 3: Prepare the shared volumes

Why every worker needs the mount

NFS server

Every worker, and the control-plane node

Step 4: Push the G&L image

Step 5: Create the namespace and pull secret

Step 6: Generate the manifests

Step 7: Complete Aveksa_System.cfg

Step 8: Correct the manifests

Showstopper 1: bind each claim to its volume

Showstopper 2: pull secret

Showstopper 3 (cluster): host names

Showstopper 4 (cluster): member list before start

RSA requirement 5: time zone

Recommended 6 to 8

Step 9: Deploy

Step 10: Verify the deployment

Step 11: Post-install tasks and backup

Day-2 operations

Stop and start

Logs

Redeploy from scratch

Newer image

System Operations Node handover

Troubleshooting

Pod Does Not Start

Pod Starts, Then Fails

G&L Runs But Has Irregularities

References