How to troubleshoot backup Failures in ExaCS

We may get a failure while running the managed backup on the ExaCS system. Let’s see How to troubleshoot backup Failures in ExaCS

Table of Contents

(1)Identification of the error

We need to first find out what is the error for the backup. We can check those in the backup log files. You can find the backup log at the below location

Log on to the host as the root user.
Check the applicable log file:

DTRS not enabled

If the failure occurred during RMAN configuration, navigate to the /var/opt/oracle/log/<database name>/bkup/ directory and check the bkup.log file.
If the failure occurred during the backup job, navigate to the /var/opt/oracle/log/<database name>/obkup/ directory and check the obkup.log file.

DTRS enabled

navigate to the /var/opt/oracle/log/<database name>/dtrs/rman

We need to check various things depending on the error given in the log files. We are discussing the various issue in below heading

(2) Database issues

(a)An improper database state or configuration can lead to failed backups.
The database must be active and running (ideally on all nodes) while the backup is in progress.
Use the following command to check the state of your database, and ensure that any problems that might have put the database in an improper state are resolved:

srvctl status database -d <db unique name> -verbose

The database must be open and running. If it is not open, please open it

If DB not running
srvctl start database -d <db unique name>
If DB is in mount state
sqlplus / as sysdba
alter database open

(b) The database could be in no archive log mode. Check the log mode

select log_mode from v$database

If it is is no archivelog mode, please enable archive log mode

srvctl stop database -d <db unique name>
Start one of the database instances in mount state:
srvctl start instance -d <db unique name> -i <instance_name> -o mount
sqlplus / as sysdba
alter database archivelog;
exit;
srvctl stop instance -d <db unique name> -i <instance_name>
srvctl start database -d <db unique name>

If the below command returns, stuck Archiver status, please resolve that and then continue

srvctl status database -d <db unique name> -verbose

You can use the below note to resolve the issue

How to Resolve ORA-00257:Archiver Error (Doc ID 2014425.1)

(d) RMAN settings changed

Following RMAN settings should not be changed in the database manually

CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 30 DAYS;
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 5 BACKUP TYPE TO COMPRESSED BACKUPSET;
CONFIGURE CHANNEL DEVICE TYPE DISK MAXPIECESIZE 2 G;
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=/var/opt/oracle/dbaas_acfs/<db_name>/opc/libopc.so, ENV=(OPC_PFILE=/var/opt/oracle/dbaas_acfs/<db_name>/opc/opc.ora)';
CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO 'SBT_TAPE';
CONFIGURE CHANNEL DEVICE TYPE DISK MAXPIECESIZE 2 G;
CONFIGURE ENCRYPTION FOR DATABASE ON;

(e) Archivelog not found

If the archivelog is deleted from ASM but not from the control file, then the backup will fail with the below error

RMAN-06059 expected archived log not found

Please run crosscheck to resolve the issue

rman target /
crosscheck archivelog all;

(f) Object store wallet file not found

RMAN backups fail when an object store wallet file is lost. The wallet file is necessary to enable connectivity to the object store.
First locate the opc file

find / -name "opc<db_name>.ora" -print /var/opt/oracle/dbaas_acfs/<db_name>

check the content of the opc file for the parameter OPC_WALLET

Confirm that the cwallet.sso file exists in the directory specified in the OPC_WALLET parameter, and confirm that the file has the correct permissions. The file permissions should have the octal value of “600” (-rw——-). Use the following command:

ls -ltr /var/opt/oracle/dbaas_acfs/<db_name>/opc/opc_wallet
-rw------- 1 oracle oinstall 0 Oct 29 01:59 cwallet.sso.lck
-rw------- 1 oracle oinstall 111231 Oct 29 01:59 cwallet.sso

(3) Host Issues

One or more of the following conditions on the database host can cause backups to fail:

If an interactive command such as oraenv, or any command that might return an error or warning message, was added to the .bash_profile file for the grid or oracle user, Database service operations like automatic backups can be interrupted and fail to complete. Check the .bash_profile file for these commands, and remove them.

Backup operations require space in the /u01 directory on the host file system. Use the df -h command on the host to check the space available for backups. If the file system has insufficient space, you can remove old log or trace files to free up space.
Customizing the site profile file ( $ORACLE_HOME/sqlplus/admin/glogin.sql ) can cause managed backups to fail in Oracle Cloud Infrastructure. In particular, interactive commands can lead to backup failures. Oracle recommends that you not modify this file for databases hosted in Oracle Cloud Infrastructure.

(4) TDE wallet issues

$ORACLE_HOME/network/admin/sqlnet.ora file must contain the ENCRYPTION_WALLET_LOCATION

The below query should return an open status for all the PDB’s

 select WRL_TYPE,WRL_PARAMETER,STATUS,WALLET_TYPE from v$encryption_wallet;

If one of the PDB is having OPEN_NO_MASTER_KEY, then below

ALTER SESSION SET CONTAINER =<pdb> ;
ADMINISTER KEY MANAGEMENT SET KEY USING TAG '<TAG>' FORCE KEYSTORE IDENTIFIED BY <keystore-password> WITH BACKUP USING

(5) Examine External Factors

Check if there were any ongoing maintenance activities in the ExaCS environment or on the cloud storage service that might have impacted the backup process.
Look for any recent changes in the network infrastructure that could affect connectivity to the backup destination.

(6)Engage Support

Oracle Support: If the issue is not resolved through these checks, consider raising a support ticket with Oracle. Provide them with all relevant logs and findings.

Community Forums: Sometimes, community forums or user groups can offer insights based on similar experiences.

Conclusion

By following these steps, you can set up a robust and automated backup system for your Oracle Exadata Cloud Service databases, leveraging the power of cloud storage for enhanced data protection and disaster recovery.

Related Articles

How to configure Automatic Managed Backup on DBCS
Oracle Database Cloud Backup Module
DBCLI commands
How to configure Automatic Managed Backup on ExaCS