Re-Linking Of Oracle Binaries

Today one of our servers crashed and the server (whole server) was restored from the backup. It took some time to be restored one by one. Meanwhile when things are set, I tried to startup the database, I got a different error that I have not seen that before.

/home/oracle /> sqlplus "/ as sysdba"

/usr/lib/pa20_64/dld.sl: Unsatisfied code symbol 'gethrtime' in load module '/mnt1/app/oracle/product/9.2.0/bin/sqlplus'.

Abort(coredump)

This surely is not due to a database fault. When searched for solutions, found that this is the problem with relinking of oracle binaries.

Relinking of oracle binaries takes place automatically when -

            - An Oracle product has been installed with an Oracle provided installer.

            - An Oracle patch set has been applied via an Oracle provided installer.

Let's look how to relink oracle binaries, if it requires at times. At times! Those times are when -

            - An OS upgrade has occurred.

            - A change has been made to the OS system libraries. This can occur during the application of an OS patch.

            - A new install failed during the relinking phase.

            - Individual Oracle executables core dump during initial startup.

            - An individual Oracle patch has been applied with explicit relink instructions or the relink is integrated into the patch install script.

Step 1 : Login as Oracle user and Set environment parameters such as $ORACLE_HOME, $ORACLE_BASE, $PATH, $LD_LIBRARY_PATH

Step 2 : Shut down all databases and listeners running on the server.

Step 3 : Relink the components as given below for different version of oracle

Oracle Version 7.3.x

For executables: oracle, exp, imp, sqlldr, tkprof

cd $ORACLE_HOME/rdbms/lib

make -f ins_rdbms.mk install

For executables: svrmgrl, svrmgrm

cd $ORACLE_HOME/svrmgr/lib

make -f ins_svrmgr.mk linstall minstall

linstall is for svrmgrl, minstall is for svrmgrm

For executables: sqlplus

cd $ORACLE_HOME/sqlplus/lib

make -f ins_sqlplus.mk install

For executables: dbsnmp, oemevent, oratclsh

cd $ORACLE_HOME/network/lib

make -f ins_agent.mk install

For executables: names, namesctl

cd $ORACLE_HOME/network/lib

make -f ins_names.mk install

For executables: tnslsnr, lsnrctl, tnsping, csmnl, trceval, trcroute

cd $ORACLE_HOME/network/lib

make -f ins_network.mk install

Oracle Version 8.0.x

For executables: oracle, exp, imp, sqlldr, tkprof, mig, dbv, orapwd, rman, svrmgrl, ogms, ogmsctl

cd $ORACLE_HOME/rdbms/lib

make -f ins_rdbms.mk install

For executables: sqlplus

cd $ORACLE_HOME/sqlplus/lib

make -f ins_sqlplus.mk install

For executables: dbsnmp, oemevent, oratclsh, libosm.so

cd $ORACLE_HOME/network/lib

make -f ins_oemagent.mk install

For executables: tnslsnr, lsnrctl, namesctl, names, osslogin, trcasst, trcroute

cd $ORACLE_HOME/network/lib

make -f ins_network.mk install

Oracle version 8.1.x (8i) or higher

A "relink" script is provided in the $ORACLE_HOME/bin directory and can be run as follows.

cd $ORACLE_HOME/bin

relink

$ relink >parameter<

accepted values for parameter: all, oracle, network, client, client_sharedlib, interMedia, precomp, utilities, oemagent

Or else you can even do individually one by one as done for version 8.0.x with a few changes stated below.

Others remain same.

For executables: names, namesctl

cd $ORACLE_HOME/network/lib

make -f ins_names.mk install

For executables: osslogin, trcasst, trcroute, onrsd, tnsping

cd $ORACLE_HOME/network/lib

make -f ins_net_client.mk install

For executables: tnslsnr, lsnrctl

cd $ORACLE_HOME/network/lib

make -f ins_net_server.mk install

Note : There are no "Relink success" messages displayed at the terminal. If the relink was failure, it will terminate with errors such as 'Fatal error', 'Ld: fatal', 'Exit Code 1.

For HP-UX servers, the relink may fail even when done manually. This is a bug and can be rectified by applying the following OS patches as per Oracle support.

For 11.0: HP/UX patches PHCO_23770, PHCO_23092 must be applied.

For 11.11: HP/UX patches PHCO_29029 and PHCO_25569 must be applied.

Recover archive gaps in standby database from primary using RMAN incremental backups

A Physical Standby database synchs with Primary by continuous apply of archive logs from a Primary Database. In case of an archive log gone missing or corrupt, We have to bring back the standby to sync with the primary.

When the logs missing or corrupt is less in number (say below 15), we can ship the logs which were missing in the standby site from the primary site (scp/sftp/ftp) and then we can register the log file in the standby so that the gap can be resolved.

Find the archives which are missing by issueing the following command.

SQL> select * from v$archive_gap;

This would give the gap sequences. Or you can use the v$managed_standby view to find where the log apply stuck.

SQL> select sequence#,process,status from v$managed_standby;

Here u can see status as wait for log for say sequence# 100 but your primary would've proceeded to sequence# 110

At primary

SQL> select max(sequence#) from v$archived_log;      ---> This would show you 110

Copy the logs to the standby site from the primary site

$ scp log_file_name_n.arc oracle@standby:/log/file/location/log_file_name_n.arc

At standby site

SQL> alter database register logfile '/log/file/location/log_file_name_n.arc';

logfile registered

Do the log file registration at the standby site until all the missing log files are registered. Now apply would take place and your standby will become sync with the primary.

This is easy process if you have missing or corrupt logs in lesser number. But when the difference is huge (say around 500 logs) this method is very time consuming and not a proper approach. Else you have to rebuild the standby database from scratch.

As an enhancement from 10g, an incremental backup created with BACKUP INCREMENTAL... FROM SCN can be used to refresh the standby database with changes at the primary database since the last SCN at Standby and then managed recovery can resume i.e. Compensate for the missing archive logs.

Let us see the steps involved.

Step 1: On the primary:

SQL> select current_scn from v$database;

CURRENT_SCN

-----------

144710998

On the standby:

SQL> select current_scn from v$database;

CURRENT_SCN

-----------

130158742

Clearly there is a difference. But this by itself does not indicate a problem; since the standby is expected to lag behind the primary (this is an asynchronous non-real time apply setup). The real question is how much it is lagging in the terms of wall clock.

To know that use the scn_to_timestamp function to translate the SCN to a timestamp:

SQL> select scn_to_timestamp(144710998) from dual;

SCN_TO_TIMESTAMP(1447102)

-------------------------------

18-AUG-11 08.54.28.000000000 AM

Run the same query to know the timestamp associated with the SCN of the standby database as well

SQL> select scn_to_timestamp(130158742) from dual;

SCN_TO_TIMESTAMP(1301571)

-------------------------------

13-AUG-11 07.19.27.000000000 PM

Note: Run it on the primary database, since it will fail in the standby in a mounted mode

This shows that the standby is four and half days lagging!

Step 2: [Standby] Stop the managed standby apply process:

SQL> alter database recover managed standby database cancel;

Database altered.

Step 3: [Standby] Shutdown the standby database

SQL> shut immediate

Step 4: [Primary] On the primary, take an incremental backup from the SCN number where the standby has been stuck:

RMAN> run {

2> allocate channel c1 type disk format '/u01/backup/%U.bkp';

3> backup incremental from scn 130158740 database;

4> }

Step 5: [Primary] On the primary, create a new standby controlfile:

SQL> alter database create standby controlfile as '/u01/backup/for_standby.ctl';

Database altered.

or

RMAN> backup current controlfile for standby format '/backups/tempfol/_%U';

Step 6: [Primary] Copy these files to standby host:

oracle@dba1 /u01/backup]$ scp * oracle@dba2:/u01/backup

Step 7: [Standby] Bring up the instance in nomount mode:

SQL> startup nomount

Step 8: [Standby] Check the location of the controlfile:

SQL> show parameter control_files

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

control_files string /u01/oradata/standby_cntfile.ctl

Step 9: [Standby] Replace the controlfile with the one you just created in primary.

 $ cp /u01/backup/for_standby.ctl /u01/oradata/standby_cntfile.ctl

or

RMAN> RESTORE STANDBY CONTROLFILE FROM '/u01/backup/_v8o069ee_1_1';

Step 10: [Standby] Mount the standby database:

SQL> alter database mount standby database;

Step 11: [Standby] Connect to RMAN. RMAN does not know about these files yet; so you must let it know – by a process called cataloging. Catalog these files:

$ rman target=/

RMAN> catalog start with '/u01/backup';

Step 12: Recover these files:

RMAN> recover database;

Step 13: After some time, the recovery fails with the message:

archive log filename=/u01/oradata/1_18108_697108460.dbf thread=1 sequence=18109

ORA-00310: archived log contains sequence 18108; sequence 18109 required

This happens because we have come to the last of the archived logs. The expected archived log with sequence# 18108 has not been generated yet.

Step 14: At this point exit RMAN and start managed recovery process:

SQL> alter database recover managed standby database disconnect from session;

Database altered.

Step 15: Check the SCN’s in primary and standby:

[Standby] SQL> select current_scn from v$database;

CURRENT_SCN

-----------

144747125

[Primary] SQL> select current_scn from v$database;

CURRENT_SCN

-----------

144747111

Now they are very close to each other. The standby has now caught up.