Failure to start up using Kubernetes / Helm Version 21.3.0 EE / XE
basilzuberi opened this issue · comments
Hi, I have found the solution already to this, but wanted to document it here for anyone else that may be looking for it.
When attempting to run the 21.3.0 EE or XE image within kubernetes using the YAML files, or even within Helm, I was getting an error stating that the control01.ctl file could not be found in the /opt/oracle/oradata/ORCLCDB/ folder, with an SPFile error.
I even logged into the pod, and could verify that nothing existed in the SID folder.
here is the Log I got:
ORACLE EDITION: ENTERPRISE
LSNRCTL for Linux: Version 21.0.0.0.0 - Production on 04-MAY-2024 02:36:31
Copyright (c) 1991, 2021, Oracle. All rights reserved.
Starting /opt/oracle/product/21c/dbhome_1/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 21.0.0.0.0 - Production
System parameter file is /opt/oracle/homes/OraDB21Home1/network/admin/listener.ora
Log messages written to /opt/oracle/diag/tnslsnr/oracle-db-0/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 21.0.0.0.0 - Production
Start Date 04-MAY-2024 02:36:31
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /opt/oracle/homes/OraDB21Home1/network/admin/listener.ora
Listener Log File /opt/oracle/diag/tnslsnr/oracle-db-0/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
The listener supports no services
The command completed successfully
[WARNING] [DBT-06208] The 'SYS' password entered does not conform to the Oracle recommended standards.
CAUSE:
a. Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9].
b.The password entered is a keyword that Oracle does not recommend to be used as password
ACTION: Specify a strong password. If required refer Oracle documentation for guidelines.
[WARNING] [DBT-06208] The 'SYSTEM' password entered does not conform to the Oracle recommended standards.
CAUSE:
a. Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9].
b.The password entered is a keyword that Oracle does not recommend to be used as password
ACTION: Specify a strong password. If required refer Oracle documentation for guidelines.
[WARNING] [DBT-06208] The 'PDBADMIN' password entered does not conform to the Oracle recommended standards.
CAUSE:
a. Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9].
b.The password entered is a keyword that Oracle does not recommend to be used as password
ACTION: Specify a strong password. If required refer Oracle documentation for guidelines.
Prepare for db operation
8% complete
Copying database files
/opt/oracle/createDB.sh: line 318: 49 Killed dbca -silent -createDatabase -enableArchive "$ENABLE_ARCHIVELOG" -archiveLogDest "$ARCHIVELOG_DIR" ${DBCA_CRED_OPTIONS} -responseFile "$ORACLE_BASE"/dbca.rsp
[ 2024-05-04 02:36:40.409 UTC ] [WARNING] [DBT-06208] The 'SYS' password entered does not conform to the Oracle recommended standards.
[ 2024-05-04 02:36:40.409 UTC ] [WARNING] [DBT-06208] The 'SYSTEM' password entered does not conform to the Oracle recommended standards.
[ 2024-05-04 02:36:40.409 UTC ] [WARNING] [DBT-06208] The 'PDBADMIN' password entered does not conform to the Oracle recommended standards.
[ 2024-05-04 02:36:45.539 UTC ] Prepare for db operation
DBCA_PROGRESS : 8%
[ 2024-05-04 02:36:45.691 UTC ] Copying database files
SQL*Plus: Release 21.0.0.0.0 - Production on Sat May 4 02:36:59 2024
Version 21.3.0.0.0
Copyright (c) 1982, 2021, Oracle. All rights reserved.
Connected to:
Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0
SQL> ALTER SYSTEM SET control_files='/opt/oracle/oradata/ORCLCDB/control01.ctl' scope=spfile
*
ERROR at line 1:
ORA-32001: write to SPFILE requested but no SPFILE is in use
SQL>
System altered.
SQL> ALTER PLUGGABLE DATABASE ORCLPDB1 SAVE STATE
*
ERROR at line 1:
ORA-01109: database not open
SQL> BEGIN DBMS_XDB_CONFIG.SETGLOBALPORTENABLED (TRUE); END;
*
ERROR at line 1:
ORA-06550: line 1, column 7:
PLS-00201: identifier 'DBMS_XDB_CONFIG.SETGLOBALPORTENABLED' must be declared
ORA-06550: line 1, column 7:
PL/SQL: Statement ignored
SQL> SQL>
Session altered.
SQL> CREATE USER OPS$oracle IDENTIFIED EXTERNALLY
*
ERROR at line 1:
ORA-01109: database not open
SQL> GRANT CREATE SESSION TO OPS$oracle
*
ERROR at line 1:
ORA-01109: database not open
SQL> GRANT SELECT ON sys.v_$pdbs TO OPS$oracle
*
ERROR at line 1:
ORA-01109: database not open
SQL> GRANT SELECT ON sys.v_$database TO OPS$oracle
*
ERROR at line 1:
ORA-01109: database not open
SQL> ALTER USER OPS$oracle SET container_data=all for sys.v_$pdbs container = current
*
ERROR at line 1:
ORA-01109: database not open
SQL> SQL> Disconnected from Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0
The Oracle base remains unchanged with value /opt/oracle
mv: cannot stat '/opt/oracle/dbs/spfileORCLCDB.ora': No such file or directory
Executing user defined scripts
/opt/oracle/runUserScripts.sh: running /opt/oracle/scripts/setup/00-allow-connections.sql
ALTER SYSTEM SET LOCAL_LISTENER='(ADDRESS=(PROTOCOL=TCP)(HOST=0.0.0.0)(PORT=1521))' SCOPE=BOTH
*
ERROR at line 1:
ORA-32001: write to SPFILE requested but no SPFILE is in use
System altered.
ERROR:
ORA-65011: Pluggable database ORCLPDB1 does not exist.
ALTER SYSTEM SET LOCAL_LISTENER='(ADDRESS=(PROTOCOL=TCP)(HOST=0.0.0.0)(PORT=1521))' SCOPE=BOTH
*
ERROR at line 1:
ORA-32001: write to SPFILE requested but no SPFILE is in use
System altered.
DONE: Executing user defined scripts
The Oracle base remains unchanged with value /opt/oracle
#####################################
########### E R R O R ###############
DATABASE SETUP WAS NOT SUCCESSFUL!
Please check output for further info!
########### E R R O R ###############
#####################################
The following output is now a tail of the alert.log:
Successful mount of redo thread 1, with mount id 201811718
2024-05-04T02:36:54.995688+00:00
Database mounted in Exclusive Mode
Lost write protection disabled
Create Relation IPS_PACKAGE_UNPACK_HISTORY
Completed: ALTER DATABASE MOUNT
2024-05-04T02:36:57.644566+00:00
Using default pga_aggregate_limit of 3840 MB
2024-05-04T02:36:59.266337+00:00
ALTER SYSTEM SET local_listener='' SCOPE=MEMORY;
2024-05-04T02:37:49.413501+00:00
Full restore complete of datafile 1 to datafile copy /opt/oracle/oradata/ORCLCDB/system01.dbf. Elapsed time: 0:00:50
checkpoint is 2601842
last deallocation scn is 2595942
Undo Optimization current scn is 1130003
2024-05-04T02:38:13.741108+00:00
Full restore complete of datafile 3 to datafile copy /opt/oracle/oradata/ORCLCDB/sysaux01.dbf. Elapsed time: 0:00:19
checkpoint is 2601842
last deallocation scn is 2600878
2024-05-04T02:38:20.396184+00:00
Full restore complete of datafile 4 to datafile copy /opt/oracle/oradata/ORCLCDB/undotbs01.dbf. Elapsed time: 0:00:01
checkpoint is 2601842
last deallocation scn is 3
2024-05-04T02:38:21.447710+00:00
Full restore complete of datafile 7 to datafile copy /opt/oracle/oradata/ORCLCDB/users01.dbf. Elapsed time: 0:00:00
checkpoint is 2601842
last deallocation scn is 3
2024-05-04T02:38:22.380279+00:00
Errors in file /opt/oracle/diag/rdbms/orclcdb/ORCLCDB/trace/ORCLCDB_ora_760.trc:
ORA-19625: error identifying file /ade/b/3738518557/oracle/oradata/SEEDDATA/system01.dbf
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
2024-05-04T02:38:22.387176+00:00
Switch of datafile 1 complete to datafile copy
checkpoint is 2601842
2024-05-04T02:38:22.395536+00:00
Errors in file /opt/oracle/diag/rdbms/orclcdb/ORCLCDB/trace/ORCLCDB_ora_760.trc:
ORA-19625: error identifying file /ade/b/3738518557/oracle/oradata/SEEDDATA/sysaux01.dbf
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
Switch of datafile 3 complete to datafile copy
checkpoint is 2601842
2024-05-04T02:38:22.409588+00:00
Errors in file /opt/oracle/diag/rdbms/orclcdb/ORCLCDB/trace/ORCLCDB_ora_760.trc:
ORA-19625: error identifying file /ade/b/3738518557/oracle/oradata/SEEDDATA/undotbs01.dbf
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
Switch of datafile 4 complete to datafile copy
checkpoint is 2601842
2024-05-04T02:38:22.426242+00:00
Errors in file /opt/oracle/diag/rdbms/orclcdb/ORCLCDB/trace/ORCLCDB_ora_760.trc:
ORA-19625: error identifying file /ade/b/3738518557/oracle/oradata/SEEDDATA/users01.dbf
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
Switch of datafile 7 complete to datafile copy
checkpoint is 2601842
Collapse
I stripped down to the following basic values of my kubernetes yaml file (deconstructed a helm chart to the basics to figure out what was wrong)
apiVersion: v1
kind: Pod
metadata:
name: oracle-pod
labels:
app: oracle
spec:
containers:
- name: oracle-container
image: path/to/repo/oracle:21.3.0-enterprise-edition
env:
- name: ORACLE_SID
value: "ORCLCDB"
- name: ORACLE_PDB
value: "ORCLPDB1"
resources:
requests:
memory: "3Gi"
cpu: "8"
limits:
memory: "3Gi"
cpu: "8"
volumeMounts:
- name: script-volume
mountPath: /opt/oracle/scripts/setup
volumes:
- name: script-volume
configMap:
name: oracle-startup-scripts
defaultMode: 0755
items:
- key: 00-script1.sh
path: 00-script1.sh
- key: 01-script2sh
path: 01-script2.sh
---
apiVersion: v1
kind: Service
metadata:
name: oracle-db-service
labels:
app: oracle
spec:
type: ClusterIP
ports:
- port: 1521
targetPort: db
protocol: TCP
name: db
- port: 5500
targetPort: http
protocol: TCP
name: http
---
apiVersion: v1
kind: ConfigMap
metadata:
name: oracle-startup-scripts
labels:
app: oracle
data:
00-script1.sh |
echo "hello world"
01-script2.sh |
echo "hello world2"The solution will be in the first comment below
The solution was to remove the resources section from helm, and just not have it. I'm not entirely sure why, but it clearly works now.
the following lines were omitted from my helm chart, and it works perfectly now:
resources:
requests:
memory: "3Gi"
cpu: "8"
limits:
memory: "3Gi"
cpu: "8"any insights to this are appreciated, but at least this MIGHT help someone.
@basilzuberi thanks. But clearly DB creation has failed because of your resource limits. 3Gi memory is too less for 8 cpus. It needs 16Gi at least.