How to Backup and Restore a RAC Database

This example illustrates how to backup and recover a 2-node, 10gR2 RAC cluster on

Linux. Backup and recovery operations for RAC databases are similar to that of single

instance databases, however there are some key differences – the goal of this post is to

highlight the steps required for recoverying RAC databases.

Backup the Database

Use the method you prefer. I used a NOCATALOG, disk-based backup for this example,

using the scripts found here.

Simulate the Failure

Shutdown the database and simulate a failure by dropping and/or corrupting a datafile.

# export ORACLE_SID=em1

# . oraenv

# srvctl stop database -d em

# srvctl status database -d em

Instance em1 is not running on node rac1

Instance em2 is not running on node rac2

# cd /u02/oracle/ora1/em

# rm sysaux01.dbf

Verify the Failure

# srvctl start instance -d em -i em1

PRKP-1001 : Error starting instance em1 on node rac1

CRS-0215: Could not start resource ‘ora.em.em1.inst’.

You will find something similar to the following in the instance’s alert log:

Sat May 24 16:53:47 2008

Errors in file /u02/oracle/logs/em/bdump/em1_dbw0_16947.trc:

ORA-01157: cannot identify/lock data file 3 – see DBWR trace file

ORA-01110: data file 3: ‘/u02/oracle/ora1/em/sysaux01.dbf’

ORA-27037: unable to obtain file status

Linux Error: 2: No such file or directory

Additional information: 3

ORA-1157 signalled during: ALTER DATABASE OPEN…

Sat May 24 16:53:49 2008

Shutting down instance (abort)

Restore the Database

First, take the database out of cluster mode via:

# sqlplus “/ as sysdba”

SQL*Plus: Release 10.2.0.3.0 – Production on Sat May 24 17:02:17 2008

Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.

Connected to an idle instance.

SQL> startup nomount;

ORACLE instance started.

Total System Global Area  268435456 bytes

Fixed Size                  1261300 bytes

Variable Size             251658508 bytes

Database Buffers            8388608 bytes

Redo Buffers                7127040 bytes

SQL> alter system set cluster_database=false scope=spfile sid=’*’;

System altered.

SQL> shutdown immediate;

ORA-01507: database not mounted

ORACLE instance shut down.

Then restore the database via RMAN:

# rman target=/

Recovery Manager: Release 10.2.0.3.0 – Production on Sat May 24

17:04:10 2008

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

connected to target database (not started)

RMAN> startup mount;

Oracle instance started

database mounted

Total System Global Area     268435456 bytes

Fixed Size                     1261300 bytes

Variable Size                251658508 bytes

Database Buffers               8388608 bytes

Redo Buffers                   7127040 bytes

RMAN> restore database;

Starting restore at 24-MAY-08

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: sid=154 devtype=DISK

channel ORA_DISK_1: starting datafile backupset restore

channel ORA_DISK_1: specifying datafile(s) to restore from backup set

restoring datafile 00001 to /u02/oracle/ora1/em/system01.dbf

restoring datafile 00002 to /u02/oracle/ora1/em/undotbs01.dbf

restoring datafile 00003 to /u02/oracle/ora1/em/sysaux01.dbf

restoring datafile 00004 to /u02/oracle/ora1/em/users01.dbf

restoring datafile 00005 to /u02/oracle/ora1/em/example01.dbf

restoring datafile 00006 to /u02/oracle/ora1/em/undotbs02.dbf

restoring datafile 00007 to /u02/oracle/ora2/em/mgmt.dbf

restoring datafile 00008 to /u02/oracle/ora1/em/mgmt_ecm_depot1.dbf

channel ORA_DISK_1: reading from backup piece

/u02/oracle/ora3/em/backups/0ijh6j4t_1_1

channel ORA_DISK_1: restored backup piece 1

piece handle=/u02/oracle/ora3/em/backups/0ijh6j4t_1_1

tag=FHB_EM1200805241630

channel ORA_DISK_1: restore complete, elapsed time: 00:06:36

Finished restore at 24-MAY-08

Since my database is so small, I will simply restore the entire thing, however, since you

know which datafile is missing, you could simply restore it and then recover the database

as necessary.

Recover the Database

RMAN> recover database;

Starting recover at 24-MAY-08

using channel ORA_DISK_1

starting media recovery

media recovery complete, elapsed time: 00:00:03

Finished recover at 24-MAY-08

RMAN> alter database open;

database opened

RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT

row

RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and

V$RMAN_OUTPUT rows

ORACLE error from target database:

ORA-06502: PL/SQL: numeric or value error: character to number

conversion error

Afterwards, place the database back into cluster mode and startup both instances:

# sqlplus “/ as sysdba”

SQL*Plus: Release 10.2.0.3.0 – Production on Sat May 24 17:16:36 2008

Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.

Connected to:

Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 – Production

With the Partitioning, Real Application Clusters, OLAP and Data Mining

options

SQL> alter system set cluster_database=true scope=spfile sid=’*’;

System altered.

SQL> shutdown immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

# srvctl start database -d em

[oracle@rac1 bdump]$ srvctl status database -d em

Instance em1 is running on node rac1

Instance em2 is running on node rac2

Convert 11gR2 non-RAC database to RAC database using rconfig

This post talks about the Conversion of  11gR2 non-RAC database to RAC database using rconfig.

Oracle provides 3 methods to convert non-RAC single instance database to RAC databases:

1. DBCA
2. rconfig
3. Enterprise Manager

All the 3 have their own benefits and can be used to suit one’s needs. My recent work involved the conversion of non-RAC single instance database to RAC database using rconfig, although I’ve tested all the 3 methods but concluded on rconfig.

Pre-requisites:

1. Configure Shared Storage setup ASM, NFS (NAS) or clustered storage.

2. A clustered Grid Infrastructure install with at least one Scan listener address. See

Oracle Grid Infrastructure Installation Guide 11g Release 2 (11.2) Linux

3.  rconfig imposes a restriction on the choice of listener. The listener must be the default listener, and it must run from the Grid Infrastructure home.

1
srvctl add listener -p 1522

After conversion, you can reconfigure the listener as required.

4.  Install Clustered Oracle Database Software as per documentation, this can be done by choosing the right configuration option. Refer to :

http://download.oracle.com/docs/cd/E11882_01/install.112/e10813/racinstl.htm#BABJGBHB

I’ve installed the new 11gR2 clustered ORACLE_HOME at

/u01/app/oracle/product/11.2.0/db_2

on both the nodes orarac01 and orarac02



Converting Single Instance Database using rconfig

1. As an “oracle” OS user navigate to

$ORACLE_HOME/assistants/rconfig/sampleXMLs

2. Open the sample file ConvertToRAC_AdminManaged.xml using a text editor such as vi. This XML sample file contains comment lines that provide instructions on how to edit the file to suit your site’s specific needs.

3.  Ensure you edit the xml with convert verify="ONLY"
The following are the sample entries:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
<!--Specify current OracleHome of non-rac database for SourceDBHome -->
 /u01/app/oracle/product/11.2.0/db_1
<!--Specify OracleHome where the rac database should be configured. It can be same as SourceDBHome -->
 /u01/app/oracle/product/11.2.0/db_2
<!--Specify SID of non-rac database and credential. User with sysdba role is required to perform conversion -->
...
<!--Specify the list of nodes that should have rac instances running for the Admin Managed Cluster Database. LocalNode should  be the first node in this nodelist. -->
...
<!--Specify Database Area Location to be configured for rac database.If this field is left empty, current storage will be use d for rac database. For CFS, this field will have directory path. -->
 +DATA

4.  Move the spfile to the shared location, in this case the Single Instance Database was hosted on file system, in this process we will move the datafiles from file system storage to ASM.

So create spfile in the shared disk location

1
SQL>create spfile='+DATA/TEST/spfiletest.ora' from pfile;

You can check if the file is created through “asmcmd”

5. Take a backup of existing $SOURCE_ORACLE_HOME/dbs/initTEST.ora, and create a new $SOURCE_ORACLE_HOME/dbs/initTEST.ora with the following parameter:

1
spfile='+DATA/TEST/spfiletest.ora'

6. Restart the Database

7. Now lets test if “rconfig” is ready for conversion, navigate to $ORACLE_HOME/bin and issue the following command

1
$./rconfig $ORACLE_HOME/assistants/rconfig/sampleXMLs/ConvertToRAC_AdminManaged.xml

The above command validates( as we’ve set convert=”ONLY”) if rconfig is ready for conversion. If the output throws any error, diagnose and troubleshoot to fix the issue. Refer to the following output for successful validation:

1
2
3
4
5
6
7
...
 Operation Succeeded
There is no return value for this step
..

8. Now are we are ready for conversion, edit the xml file “ConvertToRAC_AdminManaged.xml” and change:

from:

1
2
3
..
 <n:Convert verify="ONLY">
..

to

1
2
3
..
 <n:Convert verify="YES">
..

9. Perform the conversion

1
$./rconfig $ORACLE_HOME/assistants/rconfig/sampleXMLs/ConvertToRAC_AdminManaged.xml

The conversion will take some time to complete. The progress can be monitored from the logs located at $ORACLE_BASE/cfgtoollogs/rconfig

10. Once the conversion is complete you’d get a similar message in step 7.

11. Perform sanity checks and tweak the listener to suit your needs.

That sums up the procedure to convert Single Instance Oracle Database to RAC database. Please do share your thoughts and comments.

Reference:

http://download.oracle.com/docs/cd/E11882_01/install.112/e17214/cvrt2rac.htm#BABGGEGJ

RAC 11gR2 Setup requirements document

Architecture
Diagram 1
• Shown above is very basic 2-node 11gR2 RAC architecture.
• IP addresses shown are just for the sake of example purpose only.
• ‘rac-scan’ will have 3 IP address associated with it (discussed below).
Diagram 2
• Shown above is the line diagram for a 2-node 11gR2 RAC architecture.
Setup Requirements
1) Hardware
• Ensure servers run the same operating system binary
• At least 1024 x 768 display resolution, so that OUI displays correctly.
• Minimum 1 GB of space in the /tmp directory.
2) Network
a) NIC requirement
• Each node must have at least two network adapters or network interface cards
(NICs): one for the public network interface and one for the private network
interface (the interconnect).
• Network Interface Card (NIC) names must not contain “.” (DOT)
• The network interfaces must have the same name on all nodes
For example: With a two-node cluster, you cannot configure network adapters
on node1 with eth0 as the public interface, but on node2 have eth1 as the public
interface. Public interface names must be the same, so you must configure eth0 as
public on both nodes. You should configure the private interfaces on the same
network adapters as well. If eth1 is the private interface for node1, then eth1 should
be the private interface for node2.
• The use of a switch (or redundant switches) is required for the private network
(crossover cables are NOT supported)
• For the public network, each network adapter must support TCP/IP.
• For the private network, the interface must support the user datagram protocol
(UDP) using high-speed network adapters and switches that support TCP/IP
(minimum requirement 1 Gigabit Ethernet).
• IP address 127.0.0.1 should only map to localhost and/or localhost.localdomain,
not anything else.
• 127.*.*.* should not be used by any network interface.
• NIC bonding technique can be used for redundancy and high availability ( need
to look further into this area)

b) IP requirements
• A public IP address and A virtual IP (VIP) address for each node, with the
following characteristics:
a) Static IP address
b) Configured before installation for each node, and resolvable to that node before
installation
c) On the same subnet as all other public IP addresses, VIP addresses, and SCAN
addresses
• A private IP address for each node, with the following characteristics:
a) Static IP address
b) Configured before installation, but on a separate, private network, with its own
subnet, that is not resolvable except by other cluster member nodes
• A Single Client Access Name (SCAN) for the cluster, with the following
characteristics:
a) Three Static IP addresses configured on the domain name server (DNS) before
installation so that the three IP addresses are associated with the name provided
as the SCAN, and all three addresses are returned using a round-robin algorithm
by the DNS to the requestor. Also, the IP addresses should not be specifically
assigned to any of the nodes in the cluster.
b) Configured before installation in the DNS to resolve to addresses that are not
currently in use
c) The name must be 15 characters or less in length and must be resolvable
without the domain suffix
d) Given a name that does not begin with a numeral
e) On the same subnet as all other public IP addresses, VIP addresses, and SCAN
addresses
f) Conforms with the RFC 952 standard, which allows alphanumeric characters and
hyphens (“-“), but does not allow underscores (“_”).
So in all we will have 9 IP addresses configured for our 2-node 11gR2 RAC system
SCAN In brief:
We will be using Single Client Access Name (SCAN) feature of Oracle Real Application
Clusters (RAC) 11g Release 2.
• The SCAN works as a cluster alias for databases in the cluster.
• Three IP addresses are recommended considering load balancing and high
availability requirements regardless of the number of servers in the cluster.
• For each of the 3 IP addresses that the SCAN resolves to, a SCAN VIP resource
is created and a SCAN Listener (as shown below) is created during cluster
configuration.
• In our case, where we are using a 2-node-cluster (for which 3 IPs are still
recommended for simplification reasons), one server in the cluster will host two
sets of SCAN resources under normal operations.
• If the node where a SCAN VIP is running fails, the SCAN VIP and its associated
listener will failover to another node in the cluster
3) Storage
• With Grid Infrastructure 11gR2, RAW and block devices have been deprecated,
making ASM the recommended method of storing the OCR and Voting Disks.
• ASM disks will be used to store OCR, Voting disks, db files, recovery files.
• Raw or block devices can be used for ASM storage but Oracle strongly
recommend to use block devices for ASM setup in 11.2 onwards.
• ASM disks must be accessible from all the nodes in the cluster.
• Details on number and size of disks required will be provided by DBAs.
• Block devices should be owned by grid:asmadmin
Note: In any installation, non-ASM managed operating system storage
repositories are required, and are used for swap files, execution libraries, and
user file systems. The Oracle database and ASM executable files and libraries
must reside on the server’s operating system file system and cannot reside in an
ASM files.
4) Operating System/ Software requirements
1. OS groups:
a) dba
b) oinstall
c) oper
d) asmadmin
e) asmdba
f) asmoper
2. OS users:
a) oracle (oracle software owner ; primary group > oinstall ; secondary groups >
dba, asmdba, oper )
b) grid (primary group > oinstall ; secondary groups > asmadmin, asmdba,
asmoper )
Note: OS groups, group numbers and user must exist and be identical on all cluster
nodes.
3. Directory structure:
a) /u01/app owned by grid:oinstall with 775 permissions before installation,
and by root after the root.sh script is run during installation. This ownership and
permissions enables OUI to create the Oracle Inventory directory, in the path
/u01/app/oraInventory.
b) /u01 owned by grid:oinstall before installation, and by root after the root.sh
script is run during installation.
c) /u01/app/11.2.0/grid owned by grid:oinstall with 775 permissions.
These permissions are required for installation, and are changed during the
installation process.
d) /u01/app/grid owned by grid:oinstall with 775 permissions before
installation, and 755 permissions after installation.
e) /u01/app/oracle owned by oracle:oinstall with 775 permissions.
4. Ensure that OpenSSH is installed on your servers. OpenSSH is the required
SSH software.
5. The following packages (or later versions) must be installed:
asmlib rpms ( to be downloaded from http://www.oracle.com/technetwork/serverstorage/
linux/downloads/rhel5-084877.html )
( suitable for our linux kernel version 2.6.18-238.19.1.el5)
oracleasm-support-2.1.7-1.el5.x86_64.rpm
oracleasmlib-2.0.4-1.el5.x86_64.rpm
oracleasm-2.6.18-238.19.1.el5-2.0.5-1.el5.x86_64.rpm
oracleasm-2.6.18-238.el5-2.0.5-1.el5.x86_64.rpm
binutils-2.17.50.0.6
compat-libstdc++-33-3.2.3
elfutils-libelf-0.125
elfutils-libelf-devel-0.125
gcc-4.1.2
gcc-c++-4.1.2
glibc-2.5-24
glibc-common-2.5
glibc-devel-2.5
glibc-headers-2.5
ksh-20060214
libaio-0.3.106
libaio-devel-0.3.106
libgcc-4.1.2
libstdc++-4.1.2
libstdc++-devel 4.1.2
make-3.81
sysstat-7.0.2
unixODBC-2.2.11
unixODBC-devel-2.2.11
6. Set the default file mode creation mask (umask) to 022
for installation software owner user (grid, oracle).
7. DNS related entries should be correct and same in files
/etc/resolv.conf and /etc/nsswitch.conf on all the RAC nodes.
IMPORTANT- Besides the above OS requirements, it will be required to update few
root owned configuration files (for example /etc/sysctl.conf, /etc/hosts, /etc/security/limits.conf etc),
which will be taken care by DBA team as per Oracle stated requirements and for
doing this root access to server will be required.
root access will also be required by DBA team at the time of Oracle Grid
Infrastructure software installation.

Oracle RAC Overview

AGENDA

  • What is Real Application Cluster (RAC) ?
  • Why to use RAC ?
  • Single instance vs RAC
  • RAC Architecture
  • Sharing of Resources
  • Background processes
  • Internal structures and services
  • RAC Administration
  • ORACLE Clusterware
  • RAC Mechanism
  • Failover
  • Load Balancing

WHAT IS RAC ??
Real Application Clusters (RAC) introduced in oracle 9i, is a technology that enables a single database to work on multiple instances simultaneously on different nodes.

A RAC database requires three components:

  • cluster nodes
  • Shared storage
  • Oracle Clusterware

To identify RAC instance in a database the following can be used:

  • Use the DBMS_UTILITY.IS_CLUSTER_DATABASE function
  • Show parameter CLUSTER_DATABASE

WHY USE RAC ??
High Availability
Failover
Reliability
Scalability
Manageability
Recoverability
Transparency
Row Locking
Error Detection
Buffer Cache Management
Continuous Operations
Load Balancing/Sharing
Reduction in total cost of ownership

SINGLE INSTANCE ~ RAC

 RAC ARCHITECTURE:

Public Network:  It is the public IP on which listeners would be listening and clients would contact the listener on this public IP.

Private Network InterconnectIt is a network path i.e exclusively used for inter-instance communication used by the cluster and dedicated to the server nodes of a cluster. It is used for the synchronization ofresources and in some cases for the transfer of data. It has a high bandwidth and low latency.

Storage Network: It is the network which connects the instances to the database in RAC.

WHAT’S SHARED, WHAT’S NOT 
SHARED
Disk access
Resources that manage data
All instances have common data & controls files
NOT SHARED
1)  Each node has its own dedicated:
• System memory
• Operating system
• Database instance
• Application software
2)  Each instance has individual
• Log files and
• Rollback segments

 BACKGROUND PROCESSES:

1. Global Cache Service Processes (LMSn)
LMSn handles block transfers between the holding instance’s buffer cache and requesting foreground process on the requesting instance.
LMS maintains read consistency by rolling back any uncommitted transactions for blocks that are being requested by any remote instance.
Even if ’n’ value(0-9) varies depending on the amount of messaging traffic amongst nodes in the cluster, there is default, one LMS process per pair of CPUs.

2. Global Enqueue Service Monitor (LMON)
It constantly handles reconfiguration of locks and global resources when a node joins or leaves the cluster. Its services are also known as Cluster Group Services (CGS).


3. Global Enqueue Service Daemon (LMD)
It manages lock manager service requests for GCS resources and sends them to a service queue to be handled by the LMSn process. The LMD process also handles global deadlock detection and remote resource requests (remote resource requests are requests originating from another instance).

4. Lock Process (LCK)
LCK manages non-cache fusion resource requests such as library and row cache requests and lock requests that are local to the server. Because the LMS process handles the primary function of lock management, only a single LCK process exists in each instance.

5. Diagnosability Daemon (DIAG)
This background process monitors the health of the instance and captures diagnostic data about process failures within instances. The operation of this daemon is automated and updates an alert log file to record the activity that it performs.

NOTEIn RAC environment, all instances have their separate alert logs.

Global Service Daemon (GSD)
This is a component in RAC that receives requests from the SRVCTL control utility to execute administrative tasks like startup or shutdown. The command is executed locally on each node and the results are returned to SRVCTL. The GSD is installed on the nodes by default.

INTERNAL STRUCTURES AND SERVICES

Global Resource Directory (GRD)
Records current state and owner of each resource
Contains convert and write queues
Distributed across all instances in cluster
Maintained by GCS and GES

Global Cache Services (GCS)Implements cache coherency for database
Coordinates access to database blocks for instances

Global Enqueue Services (GES)
Controls access to other resources (locks) including library cache and dictionary cache
Performs deadlock detection

RAC ADMINISTRATION:
      
Parameter File

  • Using a shared SPFILE is recommended while each instance can have its own dedicated parameter file as well.
  • SPFILE should be placed on a shareable disk subsystem like a rawdevice, a clustered file system, or Automatic Storage Management (ASM).
  • Oracle instance parameters for the RAC environment can be grouped into three major categories i.e Unique, Identical,Instance-Specific Parameters.
  • If a parameter appears more than once in a parameter file, the last specified value is the effective value, unless the values are on consecutive lines—in that case values from consecutive lines are concatenated.
  • To set the value for a parameter the following syntax is used. An asterisk (*) or no value in place of an instance_nameimplies that the parameter value is valid for all the instances.      <instance_name>. <parameter_name>=<parameter_value>*.undo_management=auto
  • To set the parameter in SPFILE, the following command can be used.                                                 alter system set <parameter>=<value>scope=<memory/spfile/both>comment=<‘comments’>deferred sid=<sid, *>
  • When DBCA is used to create an RAC database, by default it creates an SPFILE on the shared disk subsystem used. Else it can also be created manually.

Undo Management

  • Oracle stores original values of the data called Before imagein undo segments to provide read consistency and to roll back uncommitted transactions.
  • In the Oracle RAC environment, each instance stores transaction undo data in its dedicated undo tablespace. For this we must set undo_tablespace for individual instances as follows.      prod1.undo_tablespace= undo_tbs1    prod2.undo_tablespace=undo_tbs2
  • Instances in RAC can use either automatic or manual undo management, but the parameter undo_managementhas to be the same across all the instances.

To increase the size of an undo tablespace, either of the following can be used.
1. Add another database to undo tablespace.
2. Increase the size of the existing datafile(s) belonging to the undo tablespace. 

  While using Manual Undo Management following considerations aremade.

1. Use manual undo management only if you have very good reason for not using automatic undo management.
2.Do not create other objects such as tables, indexes, and so on in the tablespace used for rollback segments.
3.Create one rollback segment for every four concurrent transactions.
Temporary Tablespace
  • In an RAC environment, all instance share the same temporary tablespace with each instance creating a temporary segment in the tablespace it is using. The size should be at least equal to the concurrent maximum requirement of all the instances.
  • The default temporary tablespace cannot be dropped or taken offline; however it can be changed followed by dropping or taking offline the original default temporary tablespace.
  • To determine the temporary tablespace used by each instance GV$SORT_SEGMENTand GV$TEMPSEG_USAGE views can be queried on INST_IDcolumn basis.
  • In a temporary tablespace group, an user will always use the same assigned temporary tablespace irrespective of the instance being used. Aninstance can reclaim the space used by other instances’temporary segments in that tablespace if required for large sorts.

Online Redologs

  • In an RAC environment, each instance has its own set of online redologfiles and redologscalled Thread.
  • The size of an online redologis independent of other instances’redologsizes and is determined by the local instance’s workload and backup and recovery considerations.
  • Each instance has exclusive write access to its own online redologfiles, however it can read another instance’s current online redologfile to perform instance recovery if needed. Thus, an online redologneeds to be located on a shared storage device and cannot be on a local disk.
  • Similar to single instance database, views V$LOG and V$LOGFILE can be used in RAC for getting information about redo logs.

 Archive Logs

Steps to enable or disable log mode
1.Set cluster_database=false for the instance:
alter system set cluster_database=false scope=spfile sid= ‘prod1’;
2. Shut down all the instances accessing the database:
srvctl stop database -d prod
3. Mount the database using the local instance:
startup mount
4. Enable archiving:
alter database archivelog/ noarchivelog;
5. Change the parameter cluster_database=true for the instance prod1:
alter system set cluster_database=true scope=spfile sid=’prod1’;
6. Shut down the local instance:
shutdown ;
7. Bring up all the instances:
srvctl start database -d prod

Flashback Area (Enable / Disable )
1. Set cluster_database=false for the instance to perform this operation.
alter system set cluster_database=false scope=spfile sid= ‘prod1′;
2. Set DB_RECOVERY_FILE_DEST_SIZE and DB_RECOVERY_FILE_DEST. The                 DB_RECOVERY_FILE_DEST parameter should point to a shareable disk subsystem. alter system set db_recovery_file_dest_size=200m scope=spfile; alter system set db_recovery_file_dest=’/ocfs2/flashback’ scope=spfile;
3. Shut down all instances accessing the database:
srvctl stop database -d prod
4. Mount the database using the local instance:
startup mount
5. Enable the flashback by issuing the following command:
alter database flashback on/off;
6. Change back the parameter to cluster_database=true for the instance prod1:
alter system set cluster_database=true scope=spfile sid=’prod1′
7. Shut down the instance:
shutdown;
8. Start all the instances:
srvctl start database -d prod

SRVCTL Utility

  • To start all instances associated with a database the following command can be executed from any of the nodes. The command also starts listeners on each node if not already running.

         srvctl start database -d <DB Name>

  • Similarly, to shut down all instances associated with the database stop command can be used which does not stop listeners, as they mightbe serving other database instances running on the same machine.

              srvctl stop database –d <DB Name>

  • Options specified by -o are directly passed on to SQL *Plus as command-line options for start/stop commands.

srvctl stop database -d <DB Name> -o immediate
srvctl start database -d <DB Name> -o force

  • To perform operation on individual instance level, -I option can be used.

           srvctl stop database -d <DB Name> -i instance <IN name>

ORACLE CLUSTERWARE

  • It is a cluster manager integrated in the Database to handle the cluster including node membership, group services, global resource management, and high availability functions. It can also be used with non-cluster database.

  • Names as it has evolved –

                     1.Oracle Cluster Management Services (OCMS) – 9.0.1 and 9.2
2.Cluster Ready Services (CRS) ( Generic, portable cluster manager) – 10.1

                     3.Oracle Clusterware (CRS has been renamed) – 10.2

  • Background processes –

            1. Cluster Synchronization Service (CSS) 
            2. Cluster Ready Services (CRS) 
            3. Event Manager (EVM) 

  • To administer Clusterware, CRSCTL utility can be used i.e present in $ORA_CRS_HOME/bin.
  • Oracle Clusterware must be installed prior to installing ORACLE database. We need ROOT user during the installation process to perform various tasks requiring super user privileges.
  • Being the first Oracle software to be installed on the system, it is susceptible to configuration errors. So it is recommended to useCluster Verification Utility (CLUVFY) on all nodes i.e introduced in Oracle 10.2 but backward compatible with 10.1.
  • Oracle Clusterware requires two files that must be located on shared storage for its operation.

             1. Oracle Cluster Registry (OCR)
2. Voting Disk

Oracle Cluster Registry (OCR)
1) Located on shared storage and in Oracle 10.2 and above can be mirrored to maximum two copies.
2) Defines cluster resources including
•  Databases and Instances ( RDBMS and ASM)
•  Services and Node Applications (VIP,ONS,GSD)
•  Listener Process

Voting Disk (Quorum Disk / File in Oracle 9i)
1)  Used to determine RAC instance membership and is located on shared storage accessible to all instances.
2)  used to determine which instance takes control of cluster in case of node failure to avoid split brain
3)  In Oracle 10.2 and above can be mirrored to only Odd number of copies (1, 3, 5 etc)

VIRTUAL IP (VIP)

  • To make the applications highly available and to eliminate SPOF,Oracle 10g introduced a new feature called CLUSTER VIPs i.e a virtual IP address different from the set of in cluster IP addresses that is used by the outside world to connect to the database.
  • A VIP name and address must be registered in the DNS along with standard static IP information. Listeners would be configured to listen on VIPs instead of the public IP.
  • When a node is down, the VIP is automatically failed over to oneof the other nodes. The node that gets the VIP will “re-ARP”to the world, indicating the new MAC address of the VIP. Clients are sent error message immediately rather than waiting for the TCP timeout value.

CACHE FUSION

1)  Underlying technology that enables RAC (starting with 9i and improved performance with 10g)
2)  Protocol that allows instances to combine their data caches intoa shared global cache.
3)  Allows any node to get the most up-to-date data information from the cache of any other node in the cluster without having to access the disk drives again.
4)  Needed when Dirty Blockof data is created

  •  Data from disk is read into memory on a node
  • Data is updated on that node
  • Data hasn’t been written to disk yet
  • Another node requests the data

 WHAT IS FAILOVER ?
1)  If a node in the shared disk cluster fails, the system dynamically redistributes the workload among the surviving cluster nodes.
2)  RAC checks to detect node and network failures. A disk-based heartbeat mechanism uses the control file to monitor node membership and the cluster interconnect is regularly checked to determine correct operation.

3)  Enhanced failover reliability in 10g with the use of Virtual IP addresses (VIPs)
4)  If one node or instance fails, node detecting failure does the following.
•  Read redo log of failed instance from last checkpoint
•  Apply redo to datafiles including undo segments (roll forward)
•  Rollback uncommitted transactions
•  Cluster is frozen during part of this process

FAST APPLICATION NOTIFICATION (FAN)

1)  FAN is a method introduced in Oracle 10.1, by which applicationscan be informed of changes in cluster status for –
•  Fast node failure detection
•  Workload balancing

2)  Advantageous by preventing applications from –
•  Waiting for TCP/IP timeouts when a node fails
•  Trying to connect to currently down database service
•  Processing data received from failed node

3)  Can be notified using –
•  Server side callouts
•  Fast Connection Failover (FCF)
•  ONS API

ORACLE NOTIFICATION SERVICE (ONS)

1)  ONS, introduced in Oracle 10.1 is a subscribe serviceused by Oracle Clusterware to propagate messages to :
•  Nodes in cluster
•  Middle-tier application servers
•  Clients

2)  ONS is automatically launched for RAC on each node as part of the Oracle Clusterware installation process.However, it can also be configured to run on nodes hosting client or mid-tier applications.

3)  It is an underlying mechanism for Fast Application Notification (FAN).

4)  ONSruns independent of Transparent Application Failover (TAF).

TRANSPARENT APPLICATION FAILOVER (TAF)

1)  TAF is a client-side feature that allows for clients to reconnect to surviving databases in the event of a failure of a database instance.
2)  Masks failures to end users; they don’t need to log back into the system
3)  Applications and users are transparently reconnected to another node
4)  Applications and queries continue uninterrupted
5)  Transactions can failover and replay
6)  Login context maintained
7)  DML transactions are rolled back
8)  Requires configuration in TNSNAMES.ORA
RAC_FAILOVER =
              (DESCRIPTION =
                (ADDRESS_LIST =
                     (FAILOVER = ON)
(ADDRESS = (PROTOCOL = TCP)(HOST = node1)(PORT = 1521)) 
(ADDRESS = (PROTOCOL = TCP)(HOST = node2)(PORT = 1521)) 
)
  (CONNECT_DATA = 
              (SERVICE_NAME = RAC)
              (SERVER = DEDICATED) (FAILOVER_MODE =(TYPE=SELECT)(METHOD=BASIC)(RETRIES=30)(DELAY=5))
)  )

Oracle RAC Components

What is RAC?

RAC stands for Real Application cluster. It is a clustering solution from Oracle Corporation that ensures high availability of databases by providing instance failover,media failover features.

Mention the Oracle RAC software components:-

Oracle RAC is composed of two or more database instances. They are composed of Memory structures and background processes same as the single instance database.Oracle RAC instances use two processes
GES(Global Enqueue Service), GCS(Global Cache Service) that enable cache fusion.

Oracle RAC instances are composed of following background processes:

ACMS—Atomic Controlfile to Memory Service (ACMS)

GTX0-j—Global Transaction Process

LMON—Global Enqueue Service Monitor

LMD—Global Enqueue Service Daemon

LMS—Global Cache Service Process

LCK0—Instance Enqueue Process

RMSn—Oracle RAC Management Processes (RMSn)

RSMN—Remote Slave Monitor

What is GRD?

GRD stands for Global Resource Directory. The GES and GCS maintains records of the statuses of each datafile and each cahed block using global resource directory.This process is referred to as cache fusion and helps in data integrity.

Give Details on Cache Fusion:-

Oracle RAC is composed of two or more instances. When a block of data is read from datafile by an instance within the cluster and another instance is in need of the same block,it is easy to get the block image from the insatnce which has the block in its SGA rather than reading from the disk. To enable inter instance communication Oracle RAC makes use of interconnects. The Global Enqueue Service(GES) monitors and Instance enqueue process manages the cache fusion.

Give Details on ACMS:-

ACMS stands for Atomic Controlfile Memory Service. In an Oracle RAC environment ACMS is an agent that ensures a distributed SGA memory update (ie) SGA updates are globally committed on success or globally aborted in event of a failure.

Give details on GTX0-j :-

The process provides transparent support for XA global transactions in a RAC environment. The database autotunes the number of these processes based on the workload of XA global transactions.

Give details on LMON:-

This process monitors global enques and resources across the cluster and performs global enqueue recovery operations. This is called as Global Enqueue Service Monitor.

Give details on LMD:-

This process is called as global enqueue service daemon. This process manages incoming remote resource requests within each instance.

Give details on LMS:-

This process is called as Global Cache service process. This process maintains statuses of datafiles and each cahed block by recording information in a Global Resource Directory (GRD). This process also controls the flow of messages to remote instances and manages global data block access and transmits block images between the buffer caches of different instances.This processing is a part of cache fusion feature.

Give details on LCK0:-

This process is called as Instance enqueue process. This process manages non-cache fusion resource requests such as library and row cache requests.

Give details on RMSn:-

This process is called as Oracle RAC management process. These pocesses perform managability tasks for Oracle RAC. Tasks include creation of resources related Oracle RAC when new instances are added to the cluster.

Give details on RSMN:-

This process is called as Remote Slave Monitor. This process manages background slave process creation and communication on remote instances. This is a background slave process.This process performs tasks on behalf of a co-ordinating process running in another instance.

What components in RAC must reside in shared storage?

All datafiles, controlfiles, SPFIles, redo log files must reside on cluster-aware shred storage.

What is the significance of using cluster-aware shared storage in an Oracle RAC environment?

All instances of an Oracle RAC can access all the datafiles, control files, SPFILE’s, redolog files when these files are hosted out of cluster-aware shared storage which are group of shared disks.

Give few examples for solutions that support cluster storage:-

ASM(automatic storage management), raw disk devices, network file system(NFS), OCFS2 and OCFS(Oracle Cluster Fie systems).

What is an interconnect network?

An interconnect network is a private network that connects all of the servers in a cluster.

The interconnect network uses a switch/multiple switches that only the nodes in the cluster can access.

How can we configure the cluster interconnect?

Configure User Datagram Protocol(UDP) on Gigabit ethernet for cluster interconnect.

On unix and linux systems we use UDP and RDS (Reliable data socket) protocols to be used by Oracle Clusterware. Windows clusters use the TCP protocol.

Can we use crossover cables with Oracle Clusterware interconnects?

No, crossover cables are not supported with Oracle Clusterware intercnects.

What is the use of cluster interconnect?

Cluster interconnect is used by the Cache fusion for inter instance communication.

How do users connect to database in an Oracle RAC environment?

Users can access a RAC database using a client/server configuration or through one or more middle tiers, with or without connection pooling. Users can use oracle services feature to connect to database.

What is the use of a service in Oracle RAC environemnt?

Applications should use the services feature to connect to the Oracle database.Services enable us to define rules and characteristics to control how users and applications connect to database instances.
What are the characteriscs controlled by Oracle services feature?

The charateristics include a unique name, workload balancing and failover options, and high availability characteristics.

Which enable the load balancing of applications in RAC?

Oracle Net Services enable the load balancing of application connections across all of the instances in an Oracle RAC database.

What is a virtual IP address or VIP?

A virtl IP address or VIP is an alternate IP address that the client connectins use instead of the standard public IP address. To configureVIP address, we need to reserve a spare IP address for each node, and the IP addresses must use the same subnet as the public network.

What is the use of VIP?

If a node fails, then the node’s VIP address fails over to another node on which the VIP address can accept TCP connections but it cannot accept Oracle connections.

Give situations under which VIP address failover happens:-

VIP addresses failover happens when the node on which the VIP address runs fails, all interfaces for the VIP address fails, all interfaces for the VIP address are disconnected from the network.

What is the significance of VIP address failover?

When a VIP address failover happens, Clients that attempt to connect to the VIP address receive a rapid connection refused error .They don’t have to wait for TCP connection timeout messages.

What are the administrative tools used for Oracle RAC environments?

Oracle RAC cluster can be administered as a single image using OEM (Enterprise Manager), SQL*PLUS, Servercontrol(SRVCTL), clusterverificationutility(cvu), DBCA, NETCA.

How do we verify that RAC instances are running?

Issue the following query from any one node connecting through SQL*PLUS.

$connect sys/sys as sysdba

SQL>select * from V$ACTIVE_INSTANCES;

The query gives the instance number under INST_NUMBER column, host_: instancename under  INST_NAME column.

What is FAN?

Fast application Notification as it abbreviates to FAN relates to the events related to instances, services and nodes. This is a notification mechanism that Oracle RAC uses to notify other processes about the configuration and service level information that includes service status changes such as, UP or DOWN events. Applications can respond to FAN events and take immediate action.

Where can we apply FAN UP and DOWN events?

FAN UP and FAN DOWN events can be applied to instances, services and nodes.

State the use of FAN events in case of a cluster configuration change?

During times of cluster configuration changes,Oracle RAC high availability framework publishes a FAN event immediately when a state change occurs in the cluster.So applications can receive FAN events and react immediately.This prevents applications from polling database and detecting a problem after such a state change.

Why should we have seperate homes for ASm instance?

It is a good practice to have ASM home seperate from the database home(ORACLE_HOME).This helps in upgrading and patching ASM and the Oracle database software independent of each other.Also,we can deinstall the Oracle database software independent of the ASM instance.

What is the advantage of using ASM?

Having ASM is the Oracle recommended storage option for RAC databases as the ASM maximizes performance by managing the storage configuration across the disks.ASM does this by distributing the database file across all of the available storage within our cluster database environment.

What is rolling upgrade?

It is a new ASM feature from Database 11g.ASM instances in Oracle database 11g release(from 11.1) can be upgraded or patched using rolling upgrade feature. This enables us to patch or upgrade ASM nodes in a clustered environment without affecting database availability.During a rolling upgrade we can maintain a functional cluster while one or more of the nodes in the cluster are running in differentsoftware versions.

Can rolling upgrade be used to upgrade from 10g to 11g database?

No, it can be used only for Oracle database 11g releases(from 11.1).

State the initialization parameters that must have same value for every instance in an Oracle RAC database:-

Some initialization parameters are critical at the database creation time and must have same values. Their value must be specified in SPFILE or PFILE for every instance. The list of parameters that must be identical on every instance are given below:

ACTIVE_INSTANCE_COUNT

ARCHIVE_LAG_TARGET

COMPATIBLE

CLUSTER_DATABASE

CLUSTER_DATABASE_INSTANCE

CONTROL_FILES

DB_BLOCK_SIZE

DB_DOMAIN

DB_FILES

DB_NAME

DB_RECOVERY_FILE_DEST

DB_RECOVERY_FILE_DEST_SIZE

DB_UNIQUE_NAME

INSTANCE_TYPE (RDBMS or ASM)

PARALLEL_MAX_SERVERS

REMOTE_LOGIN_PASSWORD_FILE

UNDO_MANAGEMENT

Can the DML_LOCKS and RESULT_CACHE_MAX_SIZE be identical on all instances?

These parameters can be identical on all instances only if these parameter values are set to zero.

What two parameters must be set at the time of starting up an ASM instance in a RAC  environment?

The parameters CLUSTER_DATABASE and INSTANCE_TYPE must be set.

Mention the components of Oracle clusterware:-

Oracle clusterware is made up of components like voting disk and Oracle Cluster Registry(OCR).

What is a CRS resource?

Oracle clusterware is used to manage high-availability operations in a cluster. Anything that Oracle Clusterware manages is known as a CRS resource.Some examples of CRS resources are database,an instance,a service,a listener,a VIP address,an application process etc.

What is the use of OCR?

Oracle clusterware manages CRS resources based on the configuration information of CRS resources stored in OCR(Oracle Cluster Registry).

How does a Oracle Clusterware manage CRS resources?

Oracle clusterware manages CRS resources based on the configuration information of CRS resources stored in OCR(Oracle Cluster Registry).

Name some Oracle clusterware tools and their uses?

OIFCFG – allocating and deallocating network interfaces

OCRCONFIG – Command-line tool for managing Oracle Cluster Registry

OCRDUMP – Identify the interconnect being used

CVU – Cluster verification utility to get status of CRS resources

What are the modes of deleting instances from ORacle Real Application cluster Databases?

We can delete instances using silent mode or interactive mode using DBCA(Database Configuration Assistant).

How do we remove ASM from a Oracle RAC environment?

We need to stop and delete the instance in the node first in interactive or silent mode.After that asm can be removed using srvctl tool as follows:

srvctl stop asm -n node_name

srvctl remove asm -n node_name

We can verify if ASM has been removed by issuing the following command:

srvctl config asm -n node_name

How do we verify that an instance has been removed from OCR after deleting an instance?

Issue the following srvctl command:

srvctl config database -d database_name

cd CRS_HOME/bin

./crs_stat

How do we verify an existing current backup of OCR?

We can verify the current backup of OCR using the following command :

Ocrconfig –showbackup

What are the performance views in an Oracle RAC environment?

We have v$ views that are instance specific. In addition we have GV$ views called as global views that has an INST_ID column of numeric data type.GV$ views obtain information from individual V$ views.

What are the types of connection load-balancing?

There are two types of connection load-balancing:server-side load balancing and clientside load balancing.

What is the differnece between server-side and client-side connection load balancing?

Client-side balancing happens at client side where load balancing is done using listener.In case of server-side load balancing listener uses a load-balancing advisory to redirect connections to the instance providing best service.

Give the usage of srvctl:-

srvctl start instance -d db_name -i “inst_name_list” [-o start_options]srvctl stop instance -d name -i “inst_name_list” [-o stop_options]srvctl stop instance -d orcl -i “orcl3,orcl4” -o immediatesrvctl start database -d name [-o start_options]srvctl stop database -d name

[-o stop_options]srvctl start database -d orcl -o mount

Oracle RAC Add New Node

Following steps needs to be done to extend or add node to existing RAC with ASM.

  1. Configure hardware on Node 3 ( rac3)
  2. Install Oracle Cluster software –services on rac3
  3. Install and create ASM instance for storage on rac3
  4. Install Oracle Software (RAC) database
  5. Configure Listener for RAC on rac3
  6. Add Database instance using DBCA

Configure hardware on Node 3 ( rac3)

– Make sure private, public network card has been configure correct IP address.

– Make sure proper entries /etc/hosts file for all three IP address

– User id has been created and base directory has correct permission.

– All the OS patches has been applied

– All the nodes are accessible using ssh without password prompt.

– Adding the public and private node names for the new node to the /etc/hosts file on the existing nodes, rac1 and rac2

– Verifying the new node can be accessed (using the ping command) from the existing nodes

– Running the following command on either docrac1 or docrac2 to verify the new node has been properly configured:

Execute following from node “rac1” or “rac2”

cluvfy stage -pre crsinst -n rac3

Install Oracle Cluster software –services on rac3

Add node or extend cluster ware – Oracle Clusterware home to the node

This will add clusterware software on “rac3” node and enable all the clusterready services

On “rac1”

– Source the Oracle Home for Oracle Cluster ware home

– Go to CRS_home/oui/bin and run the addNode.sh script.

cd /opt/oracle/crs/oui/bin

./addNode.sh

– OUI starts and first displays the Welcome window.

– The Specify Cluster Nodes to Add to Installation window appears.

– Select the node or nodes that you want to add. After selecting rac3, click Next.

– Verify the entries that OUI displays on the Summary Page and click Next.

– Run the rootaddNode.sh script from the CRS_home/install/ directory on rac1 when prompted to do so.

– Basically, this script adds the node applications of the new node to the OCR configuration.

– Run the orainstRoot.sh script on the node rac3

– Run the CRS_home/root.sh script on the node rac3 to start Oracle Clusterware on the new node.

– Add the new node’s Oracle Notification Services (ONS) configuration information to the shared Oracle Cluster Registry (OCR).

– Obtain the ONS port identifier used by the new node, which you need to know for the next step, by running the

– following command from the CRS_home/opmn/conf directory on the rac1 node:

cat ons.config

– After you locate the ONS port number for the new node, you must make sure that the ONS on rac1 can communicate with the ONS on the new node, rac3.

– To verify the installation of Oracle Clusterware on the new node, you can run the following command as the root user on the newly configured node, rac3:

CRS_home/bin/cluvfy stage -post crsinst -n docrac3 –verbose

3 Install and create ASM instance for storage on rac3

Extending the Oracle Automatic Storage Management Home Directory

Next step is to configure the shared storage for the new database instances that will be

created on new node. You should be able to access sharedas well to avoid SPOF failure needs separate ASM instance.

On rac1 , source ASM oracle home

– $ASM_HOME/oui/bin directory on rac1 and

– run the addNode.sh script.

– OUI displays the Node Selection window, select the node to be added (rac3), then click Next.

– Run the root.sh script on the new node, rac3, from the ASM home directory.

4 Oracle RAC Software Home Directory

Following process create oracle database home on rac3

On rac1

Source oracle home database

– Ensure that you have successfully installed the Oracle RAC software on at least one node in your cluster environment.

– To use these procedures as shown, your $ORACLE_HOME environment variable must identify your successfully installed Oracle RAC home directory.

– Go to the $ORACLE_HOME/oui/bin directory on rac1 and run the addNode.sh script.

– When OUI displays the Specify Cluster Nodes to Add to Installation window, select the node to be added (rac3), then click Next.

– Verify the entries that OUI displays in the Cluster Node Addition Summary window, then click Next.

– Run the root.sh script on the new node, rac3, from the $ORACLE_HOME directory on that node when OUI prompts you to do so.

– Now rac3 Oracle RAC is ready

  1. Configure Listener

Create proper new Listener on the new node using Oracle Net Configuration Assistant:

Start the Oracle Net Configuration Assistant by entering netca at the system prompt from the $ORACLE_HOME/bin directory.

  1. NETCA Select Listener configuration, and click Next.
  2. NETCA displays the Listener Configuration, Listener window.
  3. Select Add to create a new Listener, then click Next.
  4. NETCA displays the Listener Configuration, Listener Name window.
  5. Accept the default value of LISTENER for the Listener name by clicking Next.
  6. NETCA displays the Listener Configuration, Select Protocols window.
  7. Choose TCP and move it to the Selected Protocols area, then click Next.
  8. NETCA displays the Listener Configuration, TCP/IP Protocol window.
  9. Choose Use the standard port number of 1521, then click Next.
  10. NETCA displays the Real Application Clusters window.
  11. Select Cluster configuration for the type of configuration to perform, then click Next.
  12. NETCA displays the Real Application Clusters, Active Nodes window.
  1. Create Cluster Instance on the New Node

– On rac3 , make sure you have all the Oracle Home for database is properly sourced

– Start DBCA from $ORACLE_HOME/bin

Select Oracle Real Application Clusters database,

Select Instance Management, and then click Next.

DBCA displays the Instance Management window.

Select Add an Instance, then click Next.

DBCA displays the List of Cluster Databases window, which shows the databases and their current status

After you terminate your DBCA session, you should run the following command to verify the administrative privileges

on the new node and obtain detailed information about these privileges:

CRS_home/bin/cluvfy comp admprv -o db_config -d oracle_home -n docrac3 -verbose

OCR File and Voting Disk Administration by Example

Overview

Oracle Clusterware 10g, formerly known as Cluster Ready Services (CRS) is software that when installed on servers running the same operating system, enables the servers to be bound together to operate and function as a single server or cluster. This infrastructure simplifies the requirement for an Oracle Real Application Clusters (RAC) database by providing cluster software that is tightly integrated with the Oracle Database.

The Oracle Clusterware requires two critical clusterware components: a voting disk to record node membership information and the Oracle Cluster Registry (OCR) to record cluster configuration information:

Voting Disk

The voting disk is a shared partition that Oracle Clusterware uses to verify cluster node membership and status. Oracle Clusterware uses the voting disk to determine which instances are members of a cluster by way of a health check and arbitrates cluster ownership among the instances in case of network failures. The primary function of the voting disk is to manage node membership and prevent what is known as Split Brain Syndrome in which two or more instances attempt to control the RAC database. This can occur in cases where there is a break in communication between nodes through the interconnect.

The voting disk must reside on a shared disk(s) that is accessible by all of the nodes in the cluster. For high availability, Oracle recommends that you have multiple voting disks. Oracle Clusterware can be configured to maintain multiple voting disks (multiplexing) but you must have an odd number of voting disks, such as three, five, and so on. Oracle Clusterware supports a maximum of 32 voting disks. If you define a single voting disk, then you should use external mirroring to provide redundancy.

A node must be able to access more than half of the voting disks at any time. For example, if you have five voting disks configured, then a node must be able to access at least three of the voting disks at any time. If a node cannot access the minimum required number of voting disks it is evicted, or removed, from the cluster. After the cause of the failure has been corrected and access to the voting disks has been restored, you can instruct Oracle Clusterware to recover the failed node and restore it to the cluster.

What is a voting disk?

A voting disk is a file that manages information about node membership.

What are the administrative tasks involved with voting disk?

Following administrative tasks are performed with the voting disk :
1) Backing up voting disks
2) Recovering Voting disks
3) Adding voting disks
4) Deleting voting disks
5) Moving voting disks
How do we backup voting disks?
1) Oracle recommends that you back up your voting disk after the initial cluster creation and after we complete any node addition or deletion procedures.
2) First, as root user, stop Oracle Clusterware (with the crsctl stop crs command) on all nodes. Then, determine the current voting disk by issuing the following command:
         crsctl query css votedisk
3) Then, issue the dd or ocopy command to back up a voting disk, as appropriate.
Give the syntax of backing up voting disks:-
On Linux or UNIX systems:
dd if=voting_disk_name of=backup_file_name
where,
voting_disk_name is the name of the active voting disk
backup_file_name is the name of the file to which we want to back up the voting disk contents
On Windows systems, use the ocopy command:
ocopy voting_disk_name backup_file_name
What is the Oracle Recommendation for backing up voting disk?
Oracle recommends us to use the dd command to backup the voting disk with aminimum block size of 4KB.
How do you restore a voting disk?
To restore the backup of your voting disk, issue the dd or ocopy command for Linux and UNIX systems or ocopy for Windows systems respectively.
On Linux or UNIX systems:
dd if=backup_file_name of=voting_disk_name
On Windows systems, use the ocopy command:
ocopy backup_file_name voting_disk_name
where,
backup_file_name is the name of the voting disk backup file
voting_disk_name is the name of the active voting disk
How can we add and remove multiple voting disks?
If we have multiple voting disks, then we can remove the voting disks and add them back into our environment using the following commands, where path is the complete path of the location where the voting disk resides:
crsctl delete css votedisk path
crsctl add css votedisk path

How do we stop Oracle Clusterware? When do we stop it?
Before making any modification to the voting disk, as root user, stop Oracle Clusterware using the crsctl stop crs command on all nodes.
How do we add voting disk?
To add a voting disk, issue the following command as the root user, replacing the path variable with the fully qualified path name for the voting disk we want to add:
crsctl add css votedisk path -force
How do we move voting disks?
To move a voting disk, issue the following commands as the root user, replacing the path variable with the fully qualified path name for the voting disk we want to move:
crsctl delete css votedisk path -force
crsctl add css votedisk path -force

How do we remove voting disks?
To remove a voting disk, issue the following command as the root user, replacing the path variable with the fully qualified path name for the voting disk we want to remove:
crsctl delete css votedisk path -force
What should we do after modifying voting disks?
After modifying the voting disk, restart Oracle Clusterware using the crsctl start crs command on all nodes, and verify the voting disk location using the following command:
crsctl query css votedisk
When can we use -force option?
If our cluster is down, then we can include the -force option to modify the voting disk configuration, without interacting with active Oracle Clusterware daemons. However, using the -force option while any cluster node is active may corrupt our configuration.

Oracle Cluster Registry (OCR)

Maintains cluster configuration information as well as configuration information about any cluster database within the cluster. OCR is the repository of configuration information for the cluster that manages information about like the cluster node list and instance-to-node mapping information. This configuration information is used by many of the processes that make up the CRS as well as other cluster-aware applications which use this repository to share information amoung them. Some of the main components included in the OCR are:

  • Node membership information
  • Database instance, node, and other mapping information
  • ASM(if configured)
  • Application resource profiles such as VIP addresses, services, etc.
  • Service characteristics
  • Information about processes that Oracle Clusterware controls
  • Information about any third-party applications controlled by CRS (10g R2 and later)

The OCR stores configuration information in a series of key-value pairs within a directory tree structure. To view the contents of the OCR in a human-readable format, run the ocrdump command. This will dump the contents of the OCR into an ASCII text file in the current directory named OCRDUMPFILE.

The OCR must reside on a shared disk(s) that is accessible by all of the nodes in the cluster. Oracle Clusterware 10g Release 2 allows you to multiplex the OCR and Oracle recommends that you use this feature to ensure cluster high availability. Oracle Clusterware allows for a maximum of two OCR locations; one is the primary and the second is an OCR mirror. If you define a single OCR, then you should use external mirroring to provide redundancy. You can replace a failed OCR online, and you can update the OCR through supported APIs such as Enterprise Manager, the Server Control Utility (SRVCTL), or the Database Configuration Assistant (DBCA).

OCR SUMMARY

  • this is the central repository for the Oracle cluster: there we find all the information about the cluster in real-time;
  • CRS update the OCR with the information about the node failure or reconfiguration;
  • CSS update the OCR when a node is added or deleted;
  • NetCA, DBCA, SRVCTL update the OCR with the services information;
  • this is a binary file and cannot be edited;
  • the OCR information is cached on each node;
  • only one node (the master node) can update the OCR file => The master node has the OCR cache up-to-date in real time;
  • OCR file is automatically backed up in the OCR location every 4 hours:

cd $CRS_HOME/cdata/<cluster name>

ls
backup00.ocr backup01.ocr backup02.ocr day.ocr day_.ocr week.ocr

  • OCR file can be backud up manually as well running the following command:

dd if=voting_disk_name of=backup_file_name

  • the OCR files are backed up for a week and overwritten in a circulary manner;
  • because the OCR is a key component of the Oracle cluster, the OCR file must be mirrored;
  • the OCR file can be exported, imported with ocrconfig command;

There are two methods for OCR Backup (Oracle Cluster Registry)

1. Automatically generated OCR files under $CRS_HOME/cdata/crs
2. OCR export/logical backup

The Oracle Clusterware automatically creates OCR backups
Every four hours: last three copies
-At the End of the Day: last two copies
-At the end of the week: last two copies
.

To backup OCR file, copy the generated file from $CRS_HOME/cdata/crs to your backup directory (/backup/oracle).

You must run the backup as “root”.

Run the below command to take OCR export backup.
# ocrconfig -export export_file_name

Example Configuration

The example configuration used in this article consists of a two-node RAC with a clustered database named racdb.idevelopment.info running Oracle RAC 10g Release 2 on the Linux x86 platform. The two node names are racnode1 andracnode2, each hosting a single Oracle instance named racdb1 and racdb2 respectively. For a detailed guide on building the example clustered database environment, please see:

Building an Inexpensive Oracle RAC 10g Release 2 on Linux – (CentOS 5.3 / iSCSI)

The example Oracle Clusterware environment is configured with a single voting disk and a single OCR file on an OCFS2 clustered file system. Note that the voting disk is owned by the oracle user in the oinstall group with 0644 permissions while the OCR file is owned by root in the oinstall group with 0640 permissions:

[oracle@racnode1 ~]$ ls -l /u02/oradata/racdb

total 16608

-rw-r–r– 1 oracle oinstall 10240000 Aug 26 22:43 CSSFile

drwxr-xr-x 2 oracle oinstall     3896 Aug 26 23:45 dbs/

-rw-r—– 1 root   oinstall  6836224 Sep  3 23:47 OCRFile

Check Current OCR File

[oracle@racnode1 ~]$ ocrcheck

Status of Oracle Cluster Registry is as follows :

Version                  :          2

Total space (kbytes)     :     262120

Used space (kbytes)      :       4660

Available space (kbytes) :     257460

ID                       :    1331197

Device/File Name         : /u02/oradata/racdb/OCRFile

Device/File integrity check succeeded

Device/File not configured

Cluster registry integrity check succeeded

Check Current Voting Disk

[oracle@racnode1 ~]$ crsctl query css votedisk

0.     0    /u02/oradata/racdb/CSSFile

located 1 votedisk(s).

Administering the OCR File

View OCR Configuration Information

Two methods exist to verify how many OCR files are configured for the cluster as well as their location. If the cluster is up and running, use the ocrcheck utility as either the oracle or root user account:

[oracle@racnode1 ~]$ ocrcheck

Status of Oracle Cluster Registry is as follows :

Version                  :          2

Total space (kbytes)     :     262120

Used space (kbytes)      :       4660

Available space (kbytes) :     257460

ID                       :    1331197

Device/File Name         : /u02/oradata/racdb/OCRFile  <– OCR (primary)

Device/File integrity check succeeded

Device/File not configured  <– OCR Mirror (not configured)

Cluster registry integrity check succeeded

If CRS is down, you can still determine the location and number of OCR files by viewing the file ocr.loc, whose location is somewhat platform dependent. For example, on the Linux platform it is located in /etc/oracle/ocr.locwhile on Sun Solaris it is located at /var/opt/oracle/ocr.loc:

[root@racnode1 ~]# cat /etc/oracle/ocr.loc

ocrconfig_loc=/u02/oradata/racdb/OCRFile

local_only=FALSE

To view the actual contents of the OCR in a human-readable format, run the ocrdump command. This command requires the CRS stack to be running. Running the ocrdump command will dump the contents of the OCR into an ASCII text file in the current directory named OCRDUMPFILE:

[root@racnode1 ~]# ocrdump

[root@racnode1 ~]# ls -l OCRDUMPFILE

-rw-r–r– 1 root root 250304 Oct  2 22:46 OCRDUMPFILE

The ocrdump utility also allows for different output options:

#

# Write OCR contents to specified file name.

#

[root@racnode1 ~]# ocrdump /tmp/’hostname’_ocrdump_’date +%m%d%y:%H%M’

#

# Print OCR contents to the screen.

#

[root@racnode1 ~]# ocrdump -stdout -keyname SYSTEM.css

#

# Write OCR contents out to XML format.

#

[root@racnode1 ~]# ocrdump -stdout -keyname SYSTEM.css -xml > ocrdump.xml

Add an OCR File

Starting with Oracle Clusterware 10g Release 2 (10.2), users now have the ability to multiplex (mirror) the OCR. Oracle Clusterware allows for a maximum of two OCR locations; one is the primary and the second is an OCR mirror. To avoid simultaneous loss of multiple OCR files, each copy of the OCR should be placed on a shared storage device that does not share any components (controller, interconnect, and so on) with the storage devices used for the other OCR file.

Before attempting to add a mirrored OCR, determine how many OCR files are currently configured for the cluster as well as their location. If the cluster is up and running, use the ocrcheck utility as either the oracle or root user account:

[oracle@racnode1 ~]$ ocrcheck

Status of Oracle Cluster Registry is as follows :

Version                  :          2

Total space (kbytes)     :     262120

Used space (kbytes)      :       4660

Available space (kbytes) :     257460

ID                       :    1331197

Device/File Name         : /u02/oradata/racdb/OCRFile  <– OCR (primary)

Device/File integrity check succeeded

Device/File not configured  <– OCR Mirror (not configured yet)

Cluster registry integrity check succeeded

If CRS is down, you can still determine the location and number of OCR files by viewing the file ocr.loc, whose location is somewhat platform dependent. For example, on the Linux platform it is located in /etc/oracle/ocr.locwhile on Sun Solaris it is located at /var/opt/oracle/ocr.loc:

[root@racnode1 ~]# cat /etc/oracle/ocr.loc

ocrconfig_loc=/u02/oradata/racdb/OCRFile

local_only=FALSE

The results above indicate I have only one OCR file and that it is located on an OCFS2 file system. Since we are allowed a maximum of two OCR locations, I intend to create an OCR mirror and locate it on the same OCFS2 file system in the same directory as the primary OCR. Please note that I am doing this for the sake brevity. The OCR mirror should always be placed on a separate device than the primary OCR file to guard against a single point of failure.

Note that the Oracle Clusterware stack should be online and running on all nodes in the cluster while adding, replacing, or removing the OCR location and hence does not require any system downtime.

The operations performed in this section affect the OCR for the entire cluster. However, theocrconfig command cannot modify OCR configuration information for nodes that are shut down or for nodes on which Oracle Clusterware is not running. So, you should avoid shutting down nodes while modifying the OCR using the ocrconfig command. If for any reason, any of the nodes in the cluster are shut down while modifying the OCR using the ocrconfig command, you will need to perform a repair on the stopped node before it can brought online to join the cluster. Please see the section “Repair an OCR File on a Local Node” for instructions on repairing the OCR file on the affected node.

You can add an OCR mirror after an upgrade or after completing the Oracle Clusterware installation. The Oracle Universal Installer (OUI) allows you to configure either one or two OCR locations during the installation of Oracle Clusterware. If you already mirror the OCR, then you do not need to add a new OCR location; Oracle Clusterware automatically manages two OCRs when you configure normal redundancy for the OCR. As previously mentioned, Oracle RAC environments do not support more than two OCR locations; a primary OCR and a secondary (mirrored) OCR.

Run the following command to add or relocate an OCR mirror using either destination_file or disk to designate the target location of the additional OCR:

ocrconfig -replace ocrmirror <destination_file>

ocrconfig -replace ocrmirror <disk>

You must be logged in as the root user to run the ocrconfig command.
Please note that ocrconfig -replace is the only way to add/relocate OCR files/mirrors. Attempting to copy the existing OCR file to a new location and then manually adding/changing the file pointer in the ocr.loc file is not supported and will actually fail to work.

For example:

#

# Verify CRS is running on node 1.

#

[root@racnode1 ~]# crsctl check crs

CSS appears healthy

CRS appears healthy

EVM appears healthy

#

# Verify CRS is running on node 2.

#

[root@racnode2 ~]# crsctl check crs

CSS appears healthy

CRS appears healthy

EVM appears healthy

#

# Configure the shared OCR destination_file/disk before

# attempting to create the new ocrmirror on it. This example

# creates a destination_file on an OCFS2 file system.

# Failure to pre-configure the new destination_file/disk

# before attempting to run ocrconfig will result in the

# following error:

#

#     PROT-21: Invalid parameter

#

[root@racnode1 ~]# cp /dev/null /u02/oradata/racdb/OCRFile_mirror

[root@racnode1 ~]# chown root /u02/oradata/racdb/OCRFile_mirror

[root@racnode1 ~]# chgrp oinstall /u02/oradata/racdb/OCRFile_mirror

[root@racnode1 ~]# chmod 640 /u02/oradata/racdb/OCRFile_mirror

#

# Add new OCR mirror.

#

[root@racnode1 ~]# ocrconfig -replace ocrmirror /u02/oradata/racdb/OCRFile_mirror

After adding the new OCR mirror, check that it can be seen from all nodes in the cluster:

#

# Verify new OCR mirror from node 1.

#

[root@racnode1 ~]# ocrcheck

Status of Oracle Cluster Registry is as follows :

Version                  :          2

Total space (kbytes)     :     262120

Used space (kbytes)      :       4668

Available space (kbytes) :     257452

ID                       :    1331197

Device/File Name         : /u02/oradata/racdb/OCRFile

Device/File integrity check succeeded

Device/File Name         : /u02/oradata/racdb/OCRFile_mirror  <– New OCR Mirror

Device/File integrity check succeeded

Cluster registry integrity check succeeded

[root@racnode1 ~]# cat /etc/oracle/ocr.loc

#Device/file  getting replaced by device /u02/oradata/racdb/OCRFile_mirror

ocrconfig_loc=/u02/oradata/racdb/OCRFile

ocrmirrorconfig_loc=/u02/oradata/racdb/OCRFile_mirror

#

# Verify new OCR mirror from node 2.

#

[root@racnode2 ~]# ocrcheck

Status of Oracle Cluster Registry is as follows :

Version                  :          2

Total space (kbytes)     :     262120

Used space (kbytes)      :       4668

Available space (kbytes) :     257452

ID                       :    1331197

Device/File Name         : /u02/oradata/racdb/OCRFile

Device/File integrity check succeeded

Device/File Name         : /u02/oradata/racdb/OCRFile_mirror  <– New OCR Mirror

Device/File integrity check succeeded

Cluster registry integrity check succeeded

[root@racnode2 ~]# cat /etc/oracle/ocr.loc

#Device/file  getting replaced by device /u02/oradata/racdb/OCRFile_mirror

ocrconfig_loc=/u02/oradata/racdb/OCRFile

ocrmirrorconfig_loc=/u02/oradata/racdb/OCRFile_mirror

  1. To add an OCR device:

To add an OCR device, provide the full path including file name.

ocrconfig -replace ocr <filename>

To add an OCR mirror device, provide the full path including file name.

ocrconfig -replace ocrmirror <filename>

  1. To remove an OCR device:

To remove an OCR device:

ocrconfig -replace ocr

To remove an OCR mirror device

ocrconfig -replace ocrmirror

  1. To replace or move the location of an OCR device:

To replace the OCR device with <filename>, provide the full path including file name.

ocrconfig -replace ocr <filename>

To replace the OCR mirror device with <filename>, provide the full path including file name.

ocrconfig -replace ocrmirror <filename>

Backup the Voting Disk

Backing up the voting disk(s) is often performed on a regular basis by the DBA to guard the cluster against a single point of failure as the result of hardware failure or user error. Because the node membership information does not usually change, it is not a strict requirement that you back up the voting disk every day. At a minimum, however, your backup strategy should include procedures to back up all voting disks at the following times and make certain that the backups are stored in a secure location that is accessible from all nodes in the cluster in the event the voting disk(s) need to be restored:

  • After installing Oracle Clusterware
  • After adding nodes to or deleting nodes from the cluster
  • After performing voting disk add or delete operations

Oracle Clusterware 10g Release 1 (10.1) only allowed for one voting disk while Oracle Clusterware 10g Release 2 (10.2) lifted this restriction to allow for 32 voting disks. For high availability, Oracle recommends that Oracle Clusterware 10g R2 users configure multiple voting disks while keeping in mind that you must have an odd number of voting disks, such as three, five, and so on. To avoid simultaneous loss of multiple voting disks, each voting disk should be placed on a shared storage device that does not share any components (controller, interconnect, and so on) with the storage devices used for the other voting disks. If you define a single voting disk, then you should use external mirroring to provide redundancy.

To make a backup copy of the voting disk on UNIX/Linux, use the dd command:

dd if=<voting_disk_name> of=<backup_file_name> bs=<block_size>

Perform this operation on every voting disk where voting_disk_name is the name of the active voting disk (input file), backup_file_name is the name of the file to which you want to back up the voting disk contents (output file), and block_size is the value to set both the input and output block sizes. As a general rule on most platforms, including Linux and Sun, the block size for the dd command should be 4k to ensure that the backup of the voting disk gets complete blocks.

If your voting disk is stored on a raw device, use the device name in place of voting_disk_name. For example:

dd if=/dev/raw/raw3 of=/u03/crs_backup/votebackup/VotingDiskBackup.dmp bs=4k

When you use the dd command to make backups of the voting disk, the backup can be performed while the Cluster Ready Services (CRS) process is active; you do not need to stop the CRS daemons (namely, the crsd.bin process) before taking a backup of the voting disk.

The following is a working UNIX script that can be scheduled in CRON to backup the OCR File and the Voting Disk on a regular basis:

crs_components_backup_10g.ksh

For the purpose of this example, the current Oracle Clusterware environment is configured with three voting disks on an OCFS2 clustered file system that will be backed up to a local file system on one of the nodes in the cluster. For example:

#

# Query the location and number of voting disks.

#

[root@racnode1 ~]# crsctl query css votedisk

0.     0    /u02/oradata/racdb/CSSFile

1.     0    /u02/oradata/racdb/CSSFile_mirror1

2.     0    /u02/oradata/racdb/CSSFile_mirror2

#

# Backup all three voting disks.

#

[root@racnode1 ~]# dd if=/u02/oradata/racdb/CSSFile of=/u03/crs_backup/votebackup/CSSFile.bak bs=4k

2500+0 records in

2500+0 records out

10240000 bytes (10 MB) copied, 0.259862 seconds, 39.4 MB/s

[root@racnode1 ~]# dd if=/u02/oradata/racdb/CSSFile_mirror1 of=/u03/crs_backup/votebackup/CSSFile_mirror1.bak bs=4k

2500+0 records in

2500+0 records out

10240000 bytes (10 MB) copied, 0.295964 seconds, 34.6 MB/s

[root@racnode1 ~]# dd if=/u02/oradata/racdb/CSSFile_mirror2 of=/u03/crs_backup/votebackup/CSSFile_mirror2.bak bs=4k

2500+0 records in

2500+0 records out

10240000 bytes (10 MB) copied, 0.249039 seconds, 41.1 MB/s

Recover the Voting Disk

The recommended way to recover from a lost or corrupt voting disk is to restore it from a previous good backup that was taken with the dd command.

There are actually very few steps required to restore the voting disks:

  1. Shutdown CRS on all nodes in the cluster.
  2. List the current location of the voting disks.
  3. Restore each of the voting disks using thedd command from a previous good backup of the voting disks that was taken using the same dd command.
  4. Re-start CRS on all nodes in the cluster.

For example:

[root@racnode1 ~]# crsctl stop crs

[root@racnode2 ~]# crsctl stop crs

[root@racnode1 ~]# crsctl query css votedisk

[root@racnode1 ~]# # Do this for all voting disks…

[root@racnode1 ~]# dd if=<backup_voting_disk> of=<voting_disk_name> bs=4k

[root@racnode1 ~]# crsctl start crs

[root@racnode2 ~]# crsctl start crs

The following is an example of what occurs on all RAC nodes when a voting disk is destroyed. This example will manually corrupt all voting disks in the cluster. After the Oracle RAC nodes reboot from the crash, we will follow up with the steps required to restore the lost/corrupt voting disk which will make use of the voting disk backups that were created in the previous section.

Although it should go without saying, DO NOT perform this recovery scenario on a critical system like production!

First, let’s check the status of the cluster and all RAC components, list the current location of the voting disk(s), and finally list the voting disk backup that will be used to recover from:

[root@racnode1 ~]# crs_stat -t

Name           Type           Target    State     Host

————————————————————

ora.racdb.db   application    ONLINE    ONLINE    racnode2

ora….b1.inst application    ONLINE    ONLINE    racnode1

ora….b2.inst application    ONLINE    ONLINE    racnode2

ora….srvc.cs application    ONLINE    ONLINE    racnode2

ora….db1.srv application    ONLINE    ONLINE    racnode1

ora….db2.srv application    ONLINE    ONLINE    racnode2

ora….SM1.asm application    ONLINE    ONLINE    racnode1

ora….E1.lsnr application    ONLINE    ONLINE    racnode1

ora….de1.gsd application    ONLINE    ONLINE    racnode1

ora….de1.ons application    ONLINE    ONLINE    racnode1

ora….de1.vip application    ONLINE    ONLINE    racnode1

ora….SM2.asm application    ONLINE    ONLINE    racnode2

ora….E2.lsnr application    ONLINE    ONLINE    racnode2

ora….de2.gsd application    ONLINE    ONLINE    racnode2

ora….de2.ons application    ONLINE    ONLINE    racnode2

ora….de2.vip application    ONLINE    ONLINE    racnode2

[root@racnode1 ~]# crsctl query css votedisk

0.     0    /u02/oradata/racdb/CSSFile

1.     0    /u02/oradata/racdb/CSSFile_mirror1

2.     0    /u02/oradata/racdb/CSSFile_mirror2

located 3 votedisk(s).

[root@racnode1 ~]# ls -l /u03/crs_backup/votebackup

total 30048

-rw-r–r– 1 root root 10240000 Oct  8 21:24 CSSFile.bak

-rw-r–r– 1 root root 10240000 Oct  8 21:24 CSSFile_mirror1.bak

-rw-r–r– 1 root root 10240000 Oct  8 21:25 CSSFile_mirror2.bak

The next step is to simulate the corruption or loss of the voting disk(s).

Oracle RAC 10g R1 / R2 (not patched with 10.2.0.4)

If you are using Oracle RAC 10g R1 or Oracle RAC 10g R2 (not patched with 10.2.0.4), simply write zero’s to one of the voting disk:

[root@racnode1 ~]# dd if=/dev/zero of=/u02/oradata/racdb/CSSFile

Both RAC servers are now stuck and will be rebooted by CRS…

Oracle RAC 11g or higher (including Oracle RAC 10g R2 patched with 10.2.0.4)

Starting with Oracle RAC 11g R1 (including Oracle RAC 10g R2 patched with 10.2.0.4), attempting to corrupt a voting disk using dd will result in all nodes being rebooted, however, Oracle Clusterware will re-construct the corrupt voting disk and successfully bring up the RAC components. Because the voting disks do not contain persistent data, CSSD is able to fully reconstruct the voting disks so long as the cluster is running. This feature was introduced with Oracle Clusterware 11.1 and is also available with Oracle Clusterware 10.2 patched with 10.2.0.4.

This makes it a bit more difficult to corrupt a voting disk by simply writing zero’s to it. You would need to find a way to dd the voting disks and stop the cluster before any of the voting disks could be automatically recovered by CSSD. Good luck with that! To simulate the corruption (actually the loss) of the voting disk and have both nodes crash, I’m simply going to delete all of the voting disks and then manually reboot the nodes:

Delete the voting disk…

[root@racnode1 ~]# rm /u02/oradata/racdb/CSSFile

[root@racnode1 ~]# rm /u02/oradata/racdb/CSSFile_mirror1

[root@racnode1 ~]# rm /u02/oradata/racdb/CSSFile_mirror2

Reboot both nodes to simulate the crash…

[root@racnode1 ~]# reboot

[root@racnode2 ~]# reboot

After the reboot, CRS will not come up and all RAC components will be down:

[root@racnode1 ~]# crs_stat -t

CRS-0184: Cannot communicate with the CRS daemon.

[root@racnode2 ~]# crs_stat -t

CRS-0184: Cannot communicate with the CRS daemon.

Ok, let’s start the recovery process.

#

# Locate the voting disk backups that were taken in the

# previous section.

#

[root@racnode1 ~]# cd /u03/crs_backup/votebackup

[root@racnode1 votebackup]# ls -l *.bak

-rw-r–r– 1 root root 10240000 Oct  8 21:24 CSSFile.bak

-rw-r–r– 1 root root 10240000 Oct  8 21:24 CSSFile_mirror1.bak

-rw-r–r– 1 root root 10240000 Oct  8 21:25 CSSFile_mirror2.bak

#

# Recover the voting disk (or voting disks) using the same

# dd command that was used to back it up, but with the input

# file and output file in reverse.

#

[root@racnode1 ~]# dd if=/u03/crs_backup/votebackup/CSSFile.bak of=/u02/oradata/racdb/CSSFile bs=4k

2500+0 records in

2500+0 records out

10240000 bytes (10 MB) copied, 0.252425 seconds, 40.6 MB/s

[root@racnode1 ~]# dd if=/u03/crs_backup/votebackup/CSSFile_mirror1.bak of=/u02/oradata/racdb/CSSFile_mirror1 bs=4k

2500+0 records in

2500+0 records out

10240000 bytes (10 MB) copied, 0.217645 seconds, 47.0 MB/s

[root@racnode1 ~]# dd if=/u03/crs_backup/votebackup/CSSFile_mirror2.bak of=/u02/oradata/racdb/CSSFile_mirror2 bs=4k

2500+0 records in

2500+0 records out

10240000 bytes (10 MB) copied, 0.220051 seconds, 46.5 MB/s

#

# Verify the permissions on the recovered voting disk(s) are

# set appropriately.

#

[root@racnode1 ~]# chown oracle /u02/oradata/racdb/CSSFile

[root@racnode1 ~]# chgrp oinstall /u02/oradata/racdb/CSSFile

[root@racnode1 ~]# chmod 644 /u02/oradata/racdb/CSSFile

[root@racnode1 ~]# chown oracle /u02/oradata/racdb/CSSFile_mirror1

[root@racnode1 ~]# chgrp oinstall /u02/oradata/racdb/CSSFile_mirror1

[root@racnode1 ~]# chmod 644 /u02/oradata/racdb/CSSFile_mirror1

[root@racnode1 ~]# chown oracle /u02/oradata/racdb/CSSFile_mirror2

[root@racnode1 ~]# chgrp oinstall /u02/oradata/racdb/CSSFile_mirror2

[root@racnode1 ~]# chmod 644 /u02/oradata/racdb/CSSFile_mirror2

#

# With the recovered voting disk(s) in place, restart CRS

# on all Oracle RAC nodes.

#

[root@racnode1 ~]# crsctl start crs

[root@racnode2 ~]# crsctl start crs

If you have multiple voting disks, then you can remove the voting disks and add them back into your environment using the crsctl delete css votedisk path and crsctl add css votedisk path commands respectively, where path is the complete path of the location on which the voting disk resides.

After recovering the voting disk, run through several tests to verify that Oracle Clusterware is functioning correctly:

[root@racnode1 ~]# crs_stat -t

Name           Type           Target    State     Host

————————————————————

ora.racdb.db   application    ONLINE    ONLINE    racnode1

ora….b1.inst application    ONLINE    ONLINE    racnode1

ora….b2.inst application    ONLINE    ONLINE    racnode2

ora….srvc.cs application    ONLINE    ONLINE    racnode2

ora….db1.srv application    ONLINE    ONLINE    racnode1

ora….db2.srv application    ONLINE    ONLINE    racnode2

ora….SM1.asm application    ONLINE    ONLINE    racnode1

ora….E1.lsnr application    ONLINE    ONLINE    racnode1

ora….de1.gsd application    ONLINE    ONLINE    racnode1

ora….de1.ons application    ONLINE    ONLINE    racnode1

ora….de1.vip application    ONLINE    ONLINE    racnode1

ora….SM2.asm application    ONLINE    ONLINE    racnode2

ora….E2.lsnr application    ONLINE    ONLINE    racnode2

ora….de2.gsd application    ONLINE    ONLINE    racnode2

ora….de2.ons application    ONLINE    ONLINE    racnode2

ora….de2.vip application    ONLINE    ONLINE    racnode2

[root@racnode1 ~]# crsctl check crs

CSS appears healthy

CRS appears healthy

EVM appears healthy

Move the OCR

#

# The new raw storage devices for OCR should be owned by the

# root user, must be in the oinstall group, and must have

# permissions set to 640. Provide at least 280MB of disk

# space for each OCR file and verify the raw storage devices

# can be seen from all nodes in the cluster.

#

[root@racnode1 ~]# ls -l /dev/raw/raw[12]

crw-r—– 1 root oinstall 162, 1 Oct  8 21:55 /dev/raw/raw1

crw-r—– 1 root oinstall 162, 2 Oct  8 21:55 /dev/raw/raw2

[root@racnode2 ~]# ls -l /dev/raw/raw[12]

crw-r—– 1 root oinstall 162, 1 Oct  8 21:54 /dev/raw/raw1

crw-r—– 1 root oinstall 162, 2 Oct  8 21:54 /dev/raw/raw2

#

# Use the dd command to zero out the devices and make sure

# no data is written to the raw devices.

#

[root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw1

[root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw2

#

# Verify CRS is running on node 1.

#

[root@racnode1 ~]# crsctl check crs

CSS appears healthy

CRS appears healthy

EVM appears healthy

#

# Verify CRS is running on node 2.

#

[root@racnode2 ~]# crsctl check crs

CSS appears healthy

CRS appears healthy

EVM appears healthy

#

# Query the current location and number of OCR files on

# the OCFS2 file system.

#

[root@racnode1 ~]# ocrcheck

Status of Oracle Cluster Registry is as follows :

Version                  :          2

Total space (kbytes)     :     262120

Used space (kbytes)      :       4676

Available space (kbytes) :     257444

ID                       : 1513888898

Device/File Name         : /u02/oradata/racdb/OCRFile         <– OCR (primary)

Device/File integrity check succeeded

Device/File Name         : /u02/oradata/racdb/OCRFile_mirror  <– OCR (mirror)

Device/File integrity check succeeded

Cluster registry integrity check succeeded

#

# Move OCR and OCR mirror to new storage location.

#

[root@racnode1 ~]# ocrconfig -replace ocr /dev/raw/raw1

[root@racnode1 ~]# ocrconfig -replace ocrmirror /dev/raw/raw2

#

# Verify OCR relocation from node 1.

#

[root@racnode1 ~]# ocrcheck

Status of Oracle Cluster Registry is as follows :

Version                  :          2

Total space (kbytes)     :     262120

Used space (kbytes)      :       4676

Available space (kbytes) :     257444

ID                       : 1513888898

Device/File Name         : /dev/raw/raw1

Device/File integrity check succeeded

Device/File Name         : /dev/raw/raw2

Device/File integrity check succeeded

Cluster registry integrity check succeeded

#

# Verify OCR relocation from node 2.

#

[root@racnode2 ~]# ocrcheck

Status of Oracle Cluster Registry is as follows :

Version                  :          2

Total space (kbytes)     :     262120

Used space (kbytes)      :       4676

Available space (kbytes) :     257444

ID                       : 1513888898

Device/File Name         : /dev/raw/raw1

Device/File integrity check succeeded

Device/File Name         : /dev/raw/raw2

Device/File integrity check succeeded

Cluster registry integrity check succeeded

#

# Remove all deleted OCR files from the OCFS2 file system.

#

[root@racnode1 ~]# rm /u02/oradata/racdb/OCRFile

[root@racnode1 ~]# rm /u02/oradata/racdb/OCRFile_mirror

Move the Voting Disk

#

# The new raw storage devices for the voting disks should be

# owned by the oracle user, must be in the oinstall group,

# and and must have permissions set to 644. Provide at least

# 20MB of disk space for each voting disk and verify the raw

# storage devices can be seen from all nodes in the cluster.

#

[root@racnode1 ~]# ls -l /dev/raw/raw[345]

crw-r–r– 1 oracle oinstall 162, 3 Oct  8 22:44 /dev/raw/raw3

crw-r–r– 1 oracle oinstall 162, 4 Oct  8 22:45 /dev/raw/raw4

crw-r–r– 1 oracle oinstall 162, 5 Oct  9 00:22 /dev/raw/raw5

[root@racnode2 ~]# ls -l /dev/raw/raw[345]

crw-r–r– 1 oracle oinstall 162, 3 Oct  8 22:53 /dev/raw/raw3

crw-r–r– 1 oracle oinstall 162, 4 Oct  8 22:54 /dev/raw/raw4

crw-r–r– 1 oracle oinstall 162, 5 Oct  9 00:23 /dev/raw/raw5

#

# Use the dd command to zero out the devices and make sure

# no data is written to the raw devices.

#

[root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw3

[root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw4

[root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw5

#

# Query the current location and number of voting disks on

# the OCFS2 file system. There needs to be at least two

# voting disks configured before attempting to perform the

# move.

#

[root@racnode1 ~]# crsctl query css votedisk

0.     0    /u02/oradata/racdb/CSSFile

1.     0    /u02/oradata/racdb/CSSFile_mirror1

2.     0    /u02/oradata/racdb/CSSFile_mirror2

located 3 votedisk(s).

#

# Stop all application processes.

#

[root@racnode1 ~]# srvctl stop database -d racdb

[root@racnode1 ~]# srvctl stop asm -n racnode1

[root@racnode1 ~]# srvctl stop asm -n racnode2

[root@racnode1 ~]# srvctl stop nodeapps -n racnode1

[root@racnode1 ~]# srvctl stop nodeapps -n racnode2

#

# Verify all application processes are OFFLINE.

#

[root@racnode1 ~]# crs_stat -t

Name           Type           Target    State     Host

————————————————————

ora.racdb.db   application    OFFLINE   OFFLINE

ora….b1.inst application    OFFLINE   OFFLINE

ora….b2.inst application    OFFLINE   OFFLINE

ora….srvc.cs application    OFFLINE   OFFLINE

ora….db1.srv application    OFFLINE   OFFLINE

ora….db2.srv application    OFFLINE   OFFLINE

ora….SM1.asm application    OFFLINE   OFFLINE

ora….E1.lsnr application    OFFLINE   OFFLINE

ora….de1.gsd application    OFFLINE   OFFLINE

ora….de1.ons application    OFFLINE   OFFLINE

ora….de1.vip application    OFFLINE   OFFLINE

ora….SM2.asm application    OFFLINE   OFFLINE

ora….E2.lsnr application    OFFLINE   OFFLINE

ora….de2.gsd application    OFFLINE   OFFLINE

ora….de2.ons application    OFFLINE   OFFLINE

ora….de2.vip application    OFFLINE   OFFLINE

#

# Shut down CRS on node 1 and verify the CRS stack is not up.

#

[root@racnode1 ~]# crsctl stop crs

Stopping resources. This could take several minutes.

Successfully stopped CRS resources.

Stopping CSSD.

Shutting down CSS daemon.

Shutdown request successfully issued.

[root@racnode1 ~]# ps -ef | grep d.bin | grep -v grep

#

# Shut down CRS on node 2 and verify the CRS stack is not up.

#

[root@racnode2 ~]# crsctl stop crs

Stopping resources. This could take several minutes.

Successfully stopped CRS resources.

Stopping CSSD.

Shutting down CSS daemon.

Shutdown request successfully issued.

[root@racnode2 ~]# ps -ef | grep d.bin | grep -v grep

#

# Move all three voting disks to new storage location.

#

[root@racnode1 ~]# crsctl delete css votedisk /u02/oradata/racdb/CSSFile -force

successful deletion of votedisk /u02/oradata/racdb/CSSFile.

[root@racnode1 ~]# crsctl add css votedisk /dev/raw/raw3 -force

Now formatting voting disk: /dev/raw/raw3

successful addition of votedisk /dev/raw/raw3.

[root@racnode1 ~]# crsctl delete css votedisk /u02/oradata/racdb/CSSFile_mirror1 -force

successful deletion of votedisk /u02/oradata/racdb/CSSFile_mirror1.

[root@racnode1 ~]# crsctl add css votedisk /dev/raw/raw4 -force

Now formatting voting disk: /dev/raw/raw4

successful addition of votedisk /dev/raw/raw4.

[root@racnode1 ~]# crsctl delete css votedisk /u02/oradata/racdb/CSSFile_mirror2 -force

successful deletion of votedisk /u02/oradata/racdb/CSSFile_mirror2.

[root@racnode1 ~]# crsctl add css votedisk /dev/raw/raw5 -force

Now formatting voting disk: /dev/raw/raw5

successful addition of votedisk /dev/raw/raw5.

#

# Verify voting disk(s) relocation from node 1.

#

[root@racnode1 ~]# crsctl query css votedisk

0.     0    /dev/raw/raw3

1.     0    /dev/raw/raw4

2.     0    /dev/raw/raw5

located 3 votedisk(s).

#

# Verify voting disk(s) relocation from node 2.

#

[root@racnode2 ~]# crsctl query css votedisk

0.     0    /dev/raw/raw3

1.     0    /dev/raw/raw4

2.     0    /dev/raw/raw5

located 3 votedisk(s).

#

# Remove all deleted voting disk files from the OCFS2 file system.

#

[root@racnode1 ~]# rm /u02/oradata/racdb/CSSFile

[root@racnode1 ~]# rm /u02/oradata/racdb/CSSFile_mirror1

[root@racnode1 ~]# rm /u02/oradata/racdb/CSSFile_mirror2

#

# With all voting disks now located on raw storage devices,

# restart CRS on all Oracle RAC nodes.

#

[root@racnode1 ~]# crsctl start crs

[root@racnode2 ~]# crsctl start crs

Ref: http://www.idevelopment.info/data/Oracle/DBA_tips/Oracle10gRAC/CLUSTER_65.shtml

Oracle TAF

 (TAF) is a feature of the Java Database Connectivity (JDBC) Oracle Call Interface (OCI) driver. It enables the application to automatically reconnect to a database, if the database instance to which the connection is made fails. In this case, the active transactions roll back.

(TAF) is a client-side feature that allows for clients to reconnect to surviving databases in the event of a failure of a database instance. Notifications are used by the server to trigger TAF callbacks on the client-side.

TAF is configured using either client-side specified TNS connect string or using server-side service attributes. However, if both methods are used to configure TAF, the server-side service attributes will supersede the client-side settings. The server-side service attributes are the preferred way to set up TAF.

TAF can operate in one of two modes, Session Failover and Select Failover. Session Failover will recreate lost connections and sessions. Select Failover will replay queries that were in progress.

VIEW :     You can query FAILOVER_TYPEFAILOVER_METHOD, and FAILED_OVER columns in the V$SESSION view to verify that TAF is correctly configured.

When there is a failure, callback functions will be initiated on the client-side via OCI callbacks. This will work with standard OCI connections as well as Connection Pool and Session Pool connections. Please see the OCI manual for more details on callbacks, Connection Pools, and Session Pools.

TAF Database Configurations

TAF works with the following database configurations to effectively mask a database failure:
·         Oracle Real Application Clusters

·         Replicated systems

·         Standby databases

·         Single instance Oracle database

Fast Connection Failover (FCF)

Fast Connection Failover offers a driver-independent way for your Java Database Connectivity (JDBC) application to take advantage of the connection failover facilities offered by Oracle Database 10g. The Fast Connection Failover mechanism depends on the implicit connection cache feature. As a result, for Fast Connection Failover to be available, implicit connection caching must be enabled.
FCF vs TAF

TAF is always active and does not have to be set, it does not work with the OCI Connection Pool.
Oracle recommends not to use TAF and Fast Connection Failover in the same application.

TAF Implementation:

Do not set the GLOBAL_DBNAME parameter in the SID_LIST_listener_name section of the listener.ora. A statically configured global database name disables TAF.

Depending on the FAILOVER_MODE parameters, you can implement TAF in a number of ways. Oracle recommends the following methods:


·         Example: TAF with Connect-Time Failover and Client Load Balancing

·         Example: TAF Retrying a Connection

·         Example: TAF Pre-Establishing a Connection

Example: TAF with Connect-Time Failover and Client Load Balancing

Implement TAF with connect-time failover and client load balancing for multiple addresses. In the following example, Oracle Net connects randomly to one of the protocol addresses on sales1-server or sales2-server. If the instance fails after the connection, the TAF application fails over to the other node’s listener, reserving any SELECT statements in progress.

sales.us.acme.com= (DESCRIPTION=   (LOAD_BALANCE=on)   (FAILOVER=on)   (ADDRESS=        (PROTOCOL=tcp)         (HOST=sales1-server)         (PORT=1521))   (ADDRESS=        (PROTOCOL=tcp)         (HOST=sales2-server)         (PORT=1521))   (CONNECT_DATA=      (SERVICE_NAME=sales.us.acme.com)      (FAILOVER_MODE=        (TYPE=select)        (METHOD=basic))))

Example: TAF Retrying a Connection

TAF also provides the ability to automatically retry connecting if the first connection attempt fails with the RETRIES and DELAY parameters. In the following example, Oracle Net tries to reconnect to the listener on sales1-server. If the failover connection fails, Oracle Net waits 15 seconds before trying to reconnect again. Oracle Net attempts to reconnect up to 20 times.

sales.us.acme.com= (DESCRIPTION=   (ADDRESS=        (PROTOCOL=tcp)         (HOST=sales1-server)         (PORT=1521))   (CONNECT_DATA=      (SERVICE_NAME=sales.us.acme.com)      (FAILOVER_MODE=        (TYPE=select)        (METHOD=basic)        (RETRIES=20)        (DELAY=15))))

Example: TAF Pre-Establishing a Connection

A backup connection can be pre-established. The initial and backup connections must be explicitly specified. In the following example, clients that use net service name sales1.us.acme.com to connect to the listener onsales1-server are also preconnected to sales2-server. If sales1-server fails after the connection, Oracle Net fails over to sales2-server, preserving any SELECT statements in progress. Likewise, Oracle Net preconnects to sales1-server for those clients that use sales2.us.acme.com to connect to the listener on sales2-server.

sales1.us.acme.com= (DESCRIPTION=   (ADDRESS=        (PROTOCOL=tcp)         (HOST=sales1-server)         (PORT=1521))   (CONNECT_DATA=      (SERVICE_NAME=sales.us.acme.com)      (INSTANCE_NAME=sales1)      (FAILOVER_MODE=        (BACKUP=sales2.us.acme.com)        (TYPE=select)        (METHOD=preconnect)))) sales2.us.acme.com= (DESCRIPTION=   (ADDRESS=        (PROTOCOL=tcp)         (HOST=sales2-server)         (PORT=1521))   (CONNECT_DATA=      (SERVICE_NAME=sales.us.acme.com)      (INSTANCE_NAME=sales2)      (FAILOVER_MODE=        (BACKUP=sales1.us.acme.com)        (TYPE=select)        (METHOD=preconnect))))

TAF supports three types of failover types:

1.SESSION failover – If a user’s connection is lost, SESSION failover establishes a new session automatically created for the user on the backup node. This type of failover does not attempt to recover selects. This failover is ideal for OLTP (online transaction processing) systems, where transactions are small.

2. SELECT failover – If the connection is lost, Oracle Net establishes a connection to another node and re-executes the SELECT statements with cursor positioned on the row on which it was positioned prior to the failover. This mode involves overhead on the client side and Oracle NET keeps track of SELECT statements. This approach is best for data warehouse systems, where the transactions are big and complex

3. NONE — This setting is the default and failover functionality is provided. Use this setting to prevent failover.

Ref:  Doc ID 334471.1

Converting single Instance to RAC instance Using DBCA

There are different ways to convert a single instance database to RAC.

1) Manual Method

2) Using rconfig

3) Using DBCA

4) Using grid control

In this post we will see step by step method to convert a single instance database to RAC Using DBCA(Database Configuration Assitant).

In this example, I already configured two node oracle asm, clusterware homes. also i already  converted the file system from non-asm to asm

follow steps for creating a template to convert a single instance to rac instance.

[oracle@rac1 ~]$ dbca

Screen Name Response
Welcome Screen Select Single Instance database.
Operations Select Manage Templates.
Template management operation Select From an existing database(structure as well as data).
Choose the database Select Database instance: testing
Specify the name and description example: testing and description testing_database_convert_from_single_to_rac_instance_database
specify whether you want to convert 
all database
related file locations to OFA,
 
or You want to main the file
 
locations in the template.
Select Maintain the file locations.  and click FINISH.
The following operations
will be performed:
create a clone template from
the source database.
click ok
Do you want to perform
another operation
Click No to EXIT

Templates for existing single instance database has been created.

Go to $ORACLE_HOME/assistants/dbca/templates and check generated backup of single instance.

[oracle@rac1 templates]$ pwd

/u01/app/oracle/product/10.2.0/db_1/assistants/dbca/templates

[oracle@rac1 templates]$ ls -ltr

total 213584

-rw-r—–  1 oracle oinstall    12050 May 16  2005 New_Database.dbt

-rw-r—–  1 oracle oinstall     5728 Jun 30  2005 Data_Warehouse.dbc

-rw-r—–  1 oracle oinstall     5665 Jun 30  2005 Transaction_Processing.dbc

-rw-r—–  1 oracle oinstall     5608 Jun 30  2005 General_Purpose.dbc

-rwxr-xr-x  1 oracle oinstall  7061504 Jul  2  2005 Seed_Database.ctl

-rwxr-xr-x  1 oracle oinstall 93569024 Jul  2  2005 Seed_Database.dfb

-rwxr-xr-x  1 oracle oinstall   991232 Jul  2  2005 example.dmp

-rwxr-xr-x  1 oracle oinstall 13017088 Jul  2  2005 example01.dfb

-rw-r—–  1 oracle oinstall 96714752 Jul 27 13:47 testing.dfb

-rw-r—–  1 oracle oinstall  7061504 Jul 27 13:47 testing.ctl

-rw-r—–  1 oracle oinstall     5349 Jul 27 13:49 testing.dbc

Copy the pre-configured database image from rac1 machine $ORACLE_HOME/assistants/ dbca/ templates/ testing.dbf & testing.dbc   to  host rac2 environment  $ORACLE_HOME/assistants/dbca/templates/ folder

[oracle@rac1 templates]$ scp testing.dfb oracle@rac2:/u01/app/oracle/product/10.2.0/db_1/assistants/dbca/templates/testing.dfb                                   100%   92MB   2.5MB/s   00:37

[oracle@rac1 templates]$ scp testing.dbc oracle@rac2:/u01/app/oracle/product/10.2.0/db_1/assistants/dbca/templates/testing.dbc                                   100% 5349     5.2KB/s   00:00

[oracle@rac1 templates]$ scp testing.ctl oracle@rac2:/u01/app/oracle/product/10.2.0/db_1/assistants/dbca/templates/testing.ctl                                   100% 6896KB   2.2MB/s   00:03

[oracle@rac2 templates]$ pwd

/u01/app/oracle/product/10.2.0/db_1/assistants/dbca/templates

[oracle@rac2 templates]$ ls -ltr

-rw-r—–  1 oracle oinstall 96714752 Jul 27 14:24 testing.dfb

-rw-r—–  1 oracle oinstall     5349 Jul 27 14:26 testing.dbc

-rw-r—–  1 oracle oinstall  7061504 Jul 27 14:26 testing.ctl

Start the dbca to convert the single instance to rac single database.
[oracle@rac2 templates]$ dbca

Screen Name Response
Welcome to the Database
configuration assitant
Select Oracle Real Application Database
Select the operation select : create a database
Select the nodes on which you
want to create the cluster
database
Select select all.
Select a template from the
following list to create
the database:
Select the template: testing(template name created by you)
Select the
Global database name
and SID:
  Type name as you want, eg: raj.
Each oracle database may be managed
centrally using the Oracle Enterprise
Manager Grid Control or locally using
the Enterprise Manager Database Control.
Choose the management option that you
would like to use to manage this
database.
Select:  configure the database with enterprise manager
For security reasons, you must specify
password for the following user
accounts in the new database.
select: Use the same password for ALL accounts password: yourpassword confirm password: yourpassword
Select the Listeners Select:Register this database with all listeners
Select the storage mechanism Select: Automatic Storage Management(ASM)
select the disk groups to be used as
storage for the database.
Select the disk group names: eg: DATA
Specify locations for the database
files to be created. 
Select:  use Oracle-Managed Files
Choose the recovery options  uncheck all
Custom Scripts Select: No scripts to run
Oracle Database Services Details Click: Next
Initialization Parameters Click: Next
Database storage Click: Next
creation options click: Finish

[oracle@rac2 templates]$ vi /etc/oratab

edit the oratab file for instances raj as raj1 and raj2 on both nodes

[oracle@rac2 templates]$  oraenv

ORACLE_SID = [oracle] ? raj2

[oracle@rac2 templates]$ sqlplus ‘/as sysdba’

SQL*Plus: Release 10.2.0.1.0 – Production on Tue Jul 27 19:37:27 2010

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

Connected to:

Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 – Production

With the Partitioning, Real Application Clusters, OLAP and Data Mining options

SQL> select instance_name from gv$instance;

INSTANCE_NAME

—————-

raj2

raj1

Removing a NODE from RAC

  1. Delete the instance on the node to be removed
  2. Clean up ASM
  3. Remove the listener from the node to be removed
  4. Remove the node from the database
  5. Remove the node from the clusterware

You can delete the instance by using the database creation assistant (DBCA), invoke the program choose the RAC database, choose instance management and then choose delete instance, enter the sysdba user and password then choose the instance to delete.

To clean up ASM follow the below steps

  1. From node 1 run the below command to stop ASM on the node to be removed

    srvctl stop asm -n rac3
    srvctl remove asm -n rac3

  2. Now run the following on the node to be removed

      cd $ORACLE_HOME/admin
      rm -rf +ASM
    cd $ORACLE_HOME/dbs
    rm -f *ASM*

  3. Check that /etc/oratab file has no ASM entries, if so remove them

Now remove the listener for the node to be removed

  • Login as user oracle, and set your DISPLAY environment variable, then start the Network Configuration Assistant
    $ORACLE_HOME/bin/netca
  • Choose cluster management
  • Choose listener
  • Choose Remove
  • Choose the  name as LISTENER

Next we remove the node from the database

  1. Run the below script from the node to be removed

    cd $ORACLE_HOME/bin
    ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME “CLUSTER_NODES={rac3}” -local
    ./runInstaller

  2. Choose to deinstall products and select the dbhome
  3. Run the following from node 1

      cd $ORACLE_HOME/oui/bin
    ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME “CLUSTER_NODES={rac1,rac2,rac3}”

Lastly we remove the clusterware software

  1. Run the following from node 1, you obtain the port number from remoteport section in the ons.config file in $ORA_CRS_HOME/opmn/conf 
    $CRS_HOME/bin/racgons remove_config rac3:6200
  2. Run the following from the node to be removed as user root
      cd $CRS_HOME/install
    ./rootdelete.sh
  3. Now run the following from node 1 as user root, obtain the node number first
      $CRS_HOME/bin/olsnodes -n
    cd $CRS_HOME/install
    ./rootdeletenode.sh rac3,3
  4. Now run the below from the node to be removed as user oracle

    cd $CRS_HOME/oui/bin
    ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME “CLUSTER_NODES={rac3}” CRS=TRUE -local
      ./runInstaller

  5. Choose to deinstall software and remove the CRS_HOME
  6. Run the following from node as user oracle

      cd $CRS_HOME/oui/bin 
    ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME “CLUSTER_NODES={rac1,rac2,rac3}” CRS=TRUE
  7. Check that the node has been removed, the first should report “invalid node”, the second you should not see any output and the last command you should only see nodes rac1 and rac2

    srvctl status nodeapps -n rac3
    crs_stat |grep -i rac3
    olsnodes -n