We may get a failure while running the managed backup on the ExaCS system. Let’s see How to troubleshoot backup Failures in ExaCS
(1)Identification of the error
We need to first find out what is the error for the backup. We can check those in the backup log files. You can find the backup log at the below location
Log on to the host as the root user.
Check the applicable log file:
DTRS not enabled
If the failure occurred during RMAN configuration, navigate to the /var/opt/oracle/log/<database name>/bkup/ directory and check the bkup.log file.
If the failure occurred during the backup job, navigate to the /var/opt/oracle/log/<database name>/obkup/ directory and check the obkup.log file.
DTRS enabled
navigate to the /var/opt/oracle/log/<database name>/dtrs/rman
We need to check various things depending on the error given in the log files. We are discussing the various issue in below heading
(2) Database issues
(a)An improper database state or configuration can lead to failed backups.
The database must be active and running (ideally on all nodes) while the backup is in progress.
Use the following command to check the state of your database, and ensure that any problems that might have put the database in an improper state are resolved:
srvctl status database -d <db unique name> -verbose
The database must be open and running. If it is not open, please open it
If DB not running srvctl start database -d <db unique name> If DB is in mount state sqlplus / as sysdba alter database open
(b) The database could be in no archive log mode. Check the log mode
select log_mode from v$database
If it is is no archivelog mode, please enable archive log mode
srvctl stop database -d <db unique name> Start one of the database instances in mount state: srvctl start instance -d <db unique name> -i <instance_name> -o mount sqlplus / as sysdba alter database archivelog; exit; srvctl stop instance -d <db unique name> -i <instance_name> srvctl start database -d <db unique name>
(c) Database could be Archive stuck status
If the below command returns, stuck Archiver status, please resolve that and then continue
srvctl status database -d <db unique name> -verbose
You can use the below note to resolve the issue
How to Resolve ORA-00257:Archiver Error (Doc ID 2014425.1)
(d) RMAN settings changed
Following RMAN settings should not be changed in the database manually
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 30 DAYS; CONFIGURE CONTROLFILE AUTOBACKUP ON; CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 5 BACKUP TYPE TO COMPRESSED BACKUPSET; CONFIGURE CHANNEL DEVICE TYPE DISK MAXPIECESIZE 2 G; CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=/var/opt/oracle/dbaas_acfs/<db_name>/opc/libopc.so, ENV=(OPC_PFILE=/var/opt/oracle/dbaas_acfs/<db_name>/opc/opc.ora)'; CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO 'SBT_TAPE'; CONFIGURE CHANNEL DEVICE TYPE DISK MAXPIECESIZE 2 G; CONFIGURE ENCRYPTION FOR DATABASE ON;
(e) Archivelog not found
If the archivelog is deleted from ASM but not from the control file, then the backup will fail with the below error
RMAN-06059 expected archived log not found
Please run crosscheck to resolve the issue
rman target /
crosscheck archivelog all;
(f) Object store wallet file not found
RMAN backups fail when an object store wallet file is lost. The wallet file is necessary to enable connectivity to the object store.
First locate the opc file
find / -name "opc<db_name>.ora" -print /var/opt/oracle/dbaas_acfs/<db_name>
check the content of the opc file for the parameter OPC_WALLET
Confirm that the cwallet.sso file exists in the directory specified in the OPC_WALLET parameter, and confirm that the file has the correct permissions. The file permissions should have the octal value of “600” (-rw——-). Use the following command:
ls -ltr /var/opt/oracle/dbaas_acfs/<db_name>/opc/opc_wallet -rw------- 1 oracle oinstall 0 Oct 29 01:59 cwallet.sso.lck -rw------- 1 oracle oinstall 111231 Oct 29 01:59 cwallet.sso
(3) Host Issues
One or more of the following conditions on the database host can cause backups to fail:
- If an interactive command such as oraenv, or any command that might return an error or warning message, was added to the .bash_profile file for the grid or oracle user, Database service operations like automatic backups can be interrupted and fail to complete. Check the .bash_profile file for these commands, and remove them.
- Backup operations require space in the /u01 directory on the host file system. Use the df -h command on the host to check the space available for backups. If the file system has insufficient space, you can remove old log or trace files to free up space.
- Customizing the site profile file ( $ORACLE_HOME/sqlplus/admin/glogin.sql ) can cause managed backups to fail in Oracle Cloud Infrastructure. In particular, interactive commands can lead to backup failures. Oracle recommends that you not modify this file for databases hosted in Oracle Cloud Infrastructure.
(4) TDE wallet issues
$ORACLE_HOME/network/admin/sqlnet.ora file must contain the ENCRYPTION_WALLET_LOCATION
The below query should return an open status for all the PDB’s
select WRL_TYPE,WRL_PARAMETER,STATUS,WALLET_TYPE from v$encryption_wallet;If one of the PDB is having OPEN_NO_MASTER_KEY, then below
ALTER SESSION SET CONTAINER =<pdb> ; ADMINISTER KEY MANAGEMENT SET KEY USING TAG '<TAG>' FORCE KEYSTORE IDENTIFIED BY <keystore-password> WITH BACKUP USING
(5) Examine External Factors
- Check if there were any ongoing maintenance activities in the ExaCS environment or on the cloud storage service that might have impacted the backup process.
- Look for any recent changes in the network infrastructure that could affect connectivity to the backup destination.
(6)Engage Support
- Oracle Support: If the issue is not resolved through these checks, consider raising a support ticket with Oracle. Provide them with all relevant logs and findings.
- Community Forums: Sometimes, community forums or user groups can offer insights based on similar experiences.
Conclusion
By following these steps, you can set up a robust and automated backup system for your Oracle Exadata Cloud Service databases, leveraging the power of cloud storage for enhanced data protection and disaster recovery.
Related Articles
How to configure Automatic Managed Backup on DBCS
Oracle Database Cloud Backup Module
DBCLI commands
How to configure Automatic Managed Backup on ExaCS