Virtual IP in RAC

How new connection establish in Oracle RAC?
For failover configuration we should need to configure our physical ip of host name in listener configuration. Listener process is accepting new connection request and handover user process to server process or dispatcher process in Oracle.
Means using listener new connection is being established by Oracle. Once connection gets established there is no need of listener process. If new connection is trying to get session in database and listener is down then what will be happening. User process gets error message and connection fails. Because listener is down in same host or something else problem. But in Oracle RAC database environment database is in sharing mode. Oracle RAC database is shared by all connected nodes. Means more than 1 listeners are running in various nodes.
In Oracle RAC database if user process is trying to get connection with some listener and found listener is down or node is down then Oracle RAC automatically transfer this request to another listener on another node. Up to Oracle 9i we use physical IP address in listener configuration. Means if requested connection gets failed then it will be diverting to another node using physical IP address of another surviving node. But during this automatically transfer, connection should need to wait up to get error message of node down or listener down using TCP/IP connection timeout. Means session should need to wait up to getting TCP/IP timeout error dictation. Once error message is received oracle RAC automatically divert this new connection request to another surviving node.
Using physical IP address there is biggest gap to get TCP/IP timeout for failover suggestion. Session should need to wait for same timeout. High availability of Oracle RAC depends on this time wasting error message.
Why VIP (Virtual IP) needs in Oracle RAC?
From Oracle 10g, virtual IP considers to configure listener. Using virtual IP we can save our TCP/IP timeout problem because Oracle notification service maintains communication between each nodes and listeners. Once ONS found any listener down or node down, it will notify another nodes and listeners with same situation. While new connection is trying to establish connection to failure node or listener, virtual IP of failure node automatically divert to surviving node and session will be establishing in another surviving node. This process doesn’t wait for TCP/IP timeout event. Due to this new connection gets faster session establishment to another surviving nodes/listener.

Characteristic of Virtual IP in Oracle RAC:
Virtual IP (VIP) is for fast connection establishment in failover dictation. Still we can use physical IP address in Oracle 10g in listener if we have no worry for failover timing. We can change default TCP/IP timeout using operating system utilities or commands and kept smaller. But taking advantage of VIP (Virtual IP address) in Oracle 10g RAC database is advisable. There is utility also provided to configure virtual IP (vip) with RAC environment called VIPCA. Default path is $ORA_CRS_HOME/bin. During installation of Oracle RAC, it is executed.
Advantage of Virtual IP deployment in Oracle RAC:
Using VIP configuration, client can be able to get connection fast even fail over of connection request to node. Because vip automatically assign to another surviving node faster and it can’t wait for TNS timeout old fashion.
Disadvantage of Virtual IP deployment in Oracle RAC:
Some more configurations is needed in system for assign virtual IP address to nodes like in /etc/hosts and others. Some misunderstanding or confusion may occur due to multiple IP assigns in same node.
Important for VIP configuration:
The VIPs should be registered in the DNS. The VIP addresses must be on the same subnet as the public host network addresses. Each Virtual IP (VIP) configured requires an unused and resolvable IP address.
Advertisements

CRSCTL Commands

Checking CRS Status:

crsctl check crs   <<–  for the local node
crsctl check cluster  <<–  for remote nodes in the cluster

[root@node1-pub ~]#   crsctl check cluster
node1-pub    ONLINE
node2-pub    ONLINE 

Viewing Cluster name:

ocrdump -stdout -keyname SYSTEM | grep -A 1 clustername | grep ORATEXT | awk ‘{print $3}’

OR

Oracle creates a directory with the same name as Cluster under the $ORA_CRS_HOME/cdata.

[root@node1-pub ~]#   ls /u01/app/crs/cdata

Viewing No. Of Nodes configured in Cluster:

olsnodes -n -p –I   this also displays 3 IP’s

[root@node1-pub ~] #   olsnodes -n -p -i 
node1-pub       1       node1-prv       node1-vip
node2-pub       2       node2-prv       node2-vip

Viewing Votedisk Information:         crsctl query css votedisk

Viewing OCR Disk Information:   ocrcheck
Status of Oracle Cluster Registry is as follows :
Version                  :          2
Total space (kbytes)     :     262120
Used space (kbytes)      :       3848
Available space (kbytes) :     258272
ID                       :  744414276
Device/File Name         : /u02/ocfs2/ocr/OCRfile_0
Device/File integrity check succeeded
Device/File Name         : /u02/ocfs2/ocr/OCRfile_1
Device/File integrity check succeeded

Various Timeout Settings in Cluster:

crsctl get css disktimeout
crsctl get css misscount
crsctl get css  reboottime

Add/Remove OCR file in Cluster:

Removing OCR File:              ocrconfig -replace ocrmirror
ocrconfig -replace ocr

Adding OCR :

1)    Get the Current status of OCR:    ocrcheck

2)    As you can see, I only have one OCR file but not the second file which is OCR Mirror.
So, I can add second OCR (OCR Mirror) as below command.
ocrconfig -replace ocrmirror <File name>

Add/Remove Votedisk file in Cluster:
Adding Votedisk:

1)     Stop CRS on all the nodes in cluster but one.

                              crsctl stop crs

2)     Get the list of Existing Vote Disks
                      crsctl query css votedisk

3)  Backup the Vote Disk file

             dd  if=/u02/ocfs2/vote/VDFile_0  of=$ORACLE_BASE/bkp/vd/VDFile_0

4)  Add an Extra Votedisk into the Cluster:

crsctl add css votedisk /u02/ocfs2/vote/VDFile_3 <<– as oracle

5) Confirm that the file has been added successfully:

               [root@node1-pub ~] #  ls –l /u02/ocfs2/vote/VDFile_3

Removing Votedisk:         crsctl delete css votedisk /u02/ocfs2/vote/VDFile_3

Backing Up OCR


Oracle performs physical backup of OCR devices every 4 hours under the default backup direcory $ORA_CRS_HOME/cdata/<CLUSTER_NAME>
and then it rolls that forward to Daily, weekly and monthly backup.

                              ocrconfig –showbackup

Manually backing up the OCR :
ocrconfig -manualbackup      —>>>Physical Backup of OCR

You can export the contents of the OCR using below command (Logical backup).

ocrconfig -export /tmp/ocr_exp.dat -s online   —–>>>Logical Backup of OCR

Restoring OCR:       ocrconfig -restore <file name>

Locate the avialable Backups:      ocrconfig –showbackup

Perform Restore from previous Backup
[root@node2-pub ~]#   ocrconfig -restore /u01/app/crs/cdata/test-crs/week.ocr

If you have logical backup of OCR (taken using export option).

ocrconfig  -import /tmp/ocr_exp.dat

Restoring Votedisks

  • Shutdown CRS on all the nodes in Cluster.
  • Locate the current location of the Votedisks
  • Restore each of the votedisks using “dd” command from the previous good backup of Votedisk taken using the same “dd” command.
  • Start CRS on all the nodes.

crsctl stop crs
crsctl query css votedisk
dd if=<backup of Votedisk> of=<Votedisk file>     à>>>do this for all the votedisks
crsctl start crs

Single NODE RAC

What is Oracle RAC One Node?
Oracle introduced a new option called RAC One Node with the release of 11gR2 in late 2009. This option is available with Enterprise edition only. Basically, it provides a cold failover solution for Oracle databases. It’s a single instance of Oracle RAC running on one node of the cluster while the 2nd node is in a cold standby mode. If the instance fails for some reason, then RAC One Node detects it and first tries to restart the instance on the same node. The instance is relocated to the 2nd node in case there is a failure or fault in 1st node and the instance cannot be restarted on the same node. The benefit of this feature is that it automates the instance relocation without any downtime and does not need a manual intervention. It uses a technology called Omotion, which facilitates the instance migration/relocation. “RAC one” is Oracle’s answer or solution to OS clustering solution like Veritas Storage Foundation, Sun Solaris cluster, IBM HACMP, and HP Service guard etc.
Purpose
Its Oracle’s attempt to tie customers to a single vendor by eliminating the need to buy 3rd party OS cluster solutions. First, it introduced Oracle Clusterware with 10g and stopped the need to rely on 3rd party cluster software and now it intends to conquer the rest who are still using HACMP, Sun Solaris cluster etc. for cold failover.
Benefits

The Oracle RAC One node provides the following benefits:

•         Built-in cluster failover for high availability

•         Rolling patches for single instance database
•         Proactive migration / failover of the instance
•         Live migration of instances across servers
•         Online upgrade to RAC
The rolling upgrade is really useful. Upgrade to the OS, and Database can be done without any downtime unless upgrade requires some scripts to be run against the database. With RAC One Node, the DBA’s and Sys admins can be proactive and migrate/failover the instance to another node to perform any critical maintenance activity.
What it’s not suited for
According to me the RAC one node is not a viable or recommended solution in the following scenarios:
•         To load balance unlike regular RAC
•         A true high availability solution
•         As a DR solution; Data guard best suits the bill
•         For mission critical applications
Cost
It is definitely not FREE. Oracle has priced RAC one at par with Active Data Guard. The RAC One node is priced separately and costs $10,000 per processor as against $23,000 for regular RAC. The licensing cost is required for ONE node only (in a 2-node setup). RAC one node is eligible for 10-day rule, allowing a customer to migrate to another without the need to buy additional license up to 10-days in a calendar year. People arguing against paying a license fee for resources they are not using will still lament.
Conclusion
I am still not very convinced on the usefulness of RAC one node. I think customers invest in RAC for their mission critical applications and achieving high availability and load balancing at the same time. Those who don’t go for RAC rely on Data Guard and now with 11g, on Active Data Guard. So don’t see a huge requirement for RAC One except seamless failover within a data center. The licensing is a bit disappointing; they are making clients pay $10 K. Moreover RAC is free with Standard edition though one doesn’t get enterprise features and limited to 4 CPU sockets only. So, thinking RAC One will be popular among customers who are currently using standard edition and want to switch to enterprise will be wrong. However, this is still a very new feature and as more people adopt it, we will get more clarity on its’ usability. I am planning to do a POC on it and would publish the installation steps and any findings (goods things and not so good things) of my POC.

SINGLE CLIENT ACCESS NAME (SCAN)

Single Client Access Name (SCAN) is a new Oracle Real Application Clusters (RAC) 11g Release 2 feature that provides a single name for clients to access Oracle Databases running in a cluster. The benefit is that the client’s connect information does not need to change if you add or remove nodes in the cluster. Having a single name to access the cluster allows clients to use the EZConnect client and the simple JDBC thin URL to access any database running in the cluster, independently of which server(s) in the cluster the database is active. SCAN provides load balancing and failover for client connections to the database. The SCAN works as a cluster alias for databases in the cluster.

How does SCAN work ?

NETWORK REQUIREMENTS FOR USING SCAN

The SCAN is configured during the installation of Oracle Grid Infrastructure that is distributed with Oracle Database 11g Release2. Oracle Grid Infrastructure is a single Oracle Home that contains Oracle Clusterware and Oracle Automatic Storage Management. You must install Oracle Grid Infrastructure first in order to use Oracle RAC 11g Release 2. During the interview phase of the Oracle Grid Infrastructure installation, you will be prompted to provide a SCAN name. There are 2 options for defining the SCAN:

  1. Define the SCAN in your corporate DNS (Domain Name Service)
  1. Use the Grid Naming Service (GNS)

USING OPTION 1 – DEFINE THE SCAN IN YOUR CORPORATE DNS

If you choose Option 1, you must ask your network administrator to create a single name that resolves to 3 IP addresses using a round-robin algorithm. Three IP addresses are recommended considering load balancing and high availability requirements regardless of the number of servers in the cluster. The IP addresses must be on the same subnet as your public network in the cluster. The name must be 15 characters or less in length, not including the domain, and must be resolvable without the domain suffix (for example: “sales1-scan’ must be resolvable as opposed to “scan1-scan.example.com”). The IPs must not be assigned to a network interface (on the cluster), since Oracle Clusterware will take care of it.

You can check the SCAN configuration in DNS using “nslookup”. If your DNS is set up to provide round-robin access to the IPs resolved by the SCAN entry, then run the “nslookup” command at least twice to see the round-robin algorithm work. The result should be that each time, the “nslookup” would return a set of 3 IPs in a different order.

Note: If your DNS server does not return a set of 3 IPs as shown in figure 3 or does not round-robin, ask your network administrator to enable such a setup. DNS using a round-robin algorithm on its own does not ensure failover of connections. However, the Oracle Client typically handles this. It is therefore recommended that the minimum version of the client used is the Oracle Database 11g Release 2 client.

USING OPTION 2 – THE GRID NAMING SERVICE (GNS)

If you choose option 2, you only need to enter the SCAN during the interview. During the cluster configuration, three IP addresses will be acquired from a DHCP service (using GNS assumes you have a DHCP service available on your public network) to create the SCAN and name resolution for the SCAN will be provided by the GNS1.

IF YOU DO NOT HAVE A DNS SERVER AVAILABLE AT INSTALLATION TIME

Oracle Universal Installer (OUI) enforces providing a SCAN resolution during the Oracle Grid Infrastructure installation, since the SCAN concept is an essential part during the creation of Oracle RAC 11g Release 2 databases in the cluster. All Oracle Database 11g Release 2 tools used to create a database (e.g. the Database Configuration Assistant (DBCA), or the Network Configuration Assistant (NetCA)) would assume its presence. Hence, OUI will not let you continue with the installation until you have provided a suitable SCAN resolution.

However, in order to overcome the installation requirement without setting up a DNS-based SCAN resolution, you can use a hosts-file based workaround. In this case, you would use a typical hosts-file entry to resolve the SCAN to only 1 IP address and one IP address only. It is not possible to simulate the round-robin resolution that the DNS server does using a local host file. The host file look-up the OS performs will only return the first IP address that matches the name. Neither will you be able to do so in one entry (one line in the hosts-file). Thus, you will create only 1 SCAN for the cluster. (Note that you will have to change the hosts-file on all nodes in the cluster for this purpose.) This workaround might also be used when performing an upgrade from former (pre-Oracle Database 11g Release 2) releases. However, it is strongly recommended to enable the SCAN configuration as described under “Option 1” or “Option 2” above shortly after the upgrade or the initial installation. In order to make the cluster aware of the modified SCAN configuration, delete the entry in the hosts-file and then issue: “srvctl modify scan -n <scan_name>” as the root user on one node in the cluster. The scan_name provided can be the existing fully qualified name (or a new name), but should be resolved through DNS, having 3 IPs associated with it, as discussed. The remaining reconfiguration is then performed automatically.

SCAN CONFIGURATION IN THE CLUSTER

During cluster configuration, several resources are created in the cluster for SCAN. For each of the 3 IP addresses that the SCAN resolves to, a SCAN VIP resource is created and a SCAN Listener is created. The SCAN Listener is dependent on the SCAN VIP and the 3 SCAN VIPs (along with their associated listeners) will be dispersed across the cluster. This means, each pair of resources (SCAN VIP + Listener) will be started on a different server in the cluster, assuming the cluster consists of three or more nodes.

In case, a 2-node-cluster is used (for which 3 IPs are still recommended for simplification reasons), one server in the cluster will host two sets of SCAN resources under normal operations. If the node where a SCAN VIP is running fails, the SCAN VIP and its associated listener will failover to another node in the cluster. If by means of such a failure the number of available servers in the cluster becomes less than three, one server would again host two sets of SCAN resources. If a node becomes available in the cluster again, the formerly mentioned dispersion will take effect and relocate one set accordingly.

DATABASE CONFIGURATION USING SCAN

For Oracle Database 11g Release 2, SCAN is an essential part of the configuration and therefore the REMOTE_LISTENER parameter is set to the SCAN per default, assuming that the database is created using standard Oracle tools (e.g. the formerly mentioned DBCA). This allows the instances to register with the SCAN Listeners as remote listeners to provide information on what services are being provided by the instance, the current load, and a recommendation on how many incoming connections should be directed to the instance. In this context, the LOCAL_LISTENER parameter must be considered. The LOCAL_LISTENER parameter should be set to the node-VIP. If you need fully qualified domain names, ensure that LOCAL_LISTENER is set to the fully qualified domain name (e.g. node-VIP.example.com). By default, a node listener is created on each node in the cluster during cluster configuration. With Oracle Grid Infrastructure 11g Release 2 the node listener run out of the Oracle Grid Infrastructure home and listen on the node-VIP using the specified port (default port is 1521). Unlike in former database versions, it is not recommended to set your REMOTE_LISTENER parameter to a server side TNSNAMES alias that resolves the host to the SCAN (HOST=sales1-scan for example) in the address list entry, but use the simplified “SCAN:port” syntax as shown in figure 5.

[oracle@mynode] srvctl config scan_listener

SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521

SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521

SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521

[oracle@mynode] srvctl config scan

SCAN name: sales1-scan, Network: 1/133.22.67.0/255.255.255.0/

SCAN VIP name: scan1, IP: /sales1-scan.example.com/133.22.67.192

SCAN VIP name: scan2, IP: /sales1-scan.example.com/133.22.67.193

SCAN VIP name: scan3, IP: /sales1-scan.example.com/133.22.67.194

NAME                  TYPE           VALUE

———–             ———–    ———————

local_listener     string           (DESCRIPTION=

(ADDRESS_LIST=

(ADDRESS=(PROTOCOL=TCP)(HOST=133.22.67.111)(PORT=1521))))

remote_listener    string             sales1-scan.example.com:1521

Note: if you are using the easy connect naming method, you may need to modify your SQLNET.ORA to ensure that EZCONNECT is in the list when specifying the order of the naming methods used for the client name resolution lookups (the Oracle 11g Release 2 default is NAMES.DIRECTORY_PATH=(tnsnames, ldap, ezconnect))

HOW CONNECTION LOAD BALANCING WORKS USING SCAN

For clients connecting using Oracle SQL*Net 11g Release 2, three IP addresses will be received by the client by resolving the SCAN name through DNS as discussed. The client will then go through the list it receives from the DNS and try connecting through one of the IPs received. If the client receives an error, it will try the other addresses before returning an error to the user or application. This is similar to how client connection failover works in previous releases when an address list is provided in the client connection string.

When a SCAN Listener receives a connection request, the SCAN Listener will check for the least loaded instance providing the requested service. It will then re-direct the connection request to the local listener on the node where the least loaded instance is running. Subsequently, the client will be given the address of the local listener. The local listener will finally create the connection to the database instance.

Note:  This example assumes an Oracle 11g R2 client using a default TNSNAMES. ORA:

ORCLservice =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = sales1-scan.example.com)(PORT = 1521))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = MyORCLservice)

))

VERSION AND BACKWARD COMPATIBILITY

The successful use of SCAN to connect to an Oracle RAC database in the cluster depends on the ability of the client to understand and use the SCAN as well as on the correct configuration of the REMOTE_LISTENER parameter setting in the database. If the version of the Oracle Client connecting to the database as well as the Oracle Database version used are both Oracle Database 11g Release 2 and the default configuration is used as described in this paper, no changes to the system are typically required.

The same holds true, if the Oracle Client version and the version of the Oracle Database that this client is connecting to are both pre-11g Release 2 version (e.g. Oracle Database 11g Release 1 or Oracle Database 10g Release 2, or older). In this case, the pre-11g Release 2 client would use a TNS connect descriptor that resolves to the node-VIPs of the cluster, while the Oracle pre-11g Release 2 database would still use a REMOTE_LISTENER entry pointing to the node-VIPs. The disadvantage of this configuration is that SCAN would not be used and hence the clients are still exposed to changes every time the cluster changes in the backend. Similarly, if an Oracle Database 11g Release 2 is used, but the clients remain on a former version. The solution is to change the Oracle client and / or Oracle Database REMOTE_LISTENER settings accordingly. The following cases need to be considered:

NoteIf using a pre-11g Release 2 client (Oracle Database 11g Release or Oracle Database 10g Release 2, or older) you will not fully benefit from the advantages of SCAN. Reason: The Oracle Client will not be able to handle a set of three IPs returned by the DNS for SCAN. Hence, it will try to connect to only the first address returned in the list and will more or less ignore the others. If the SCAN Listener listening on this specific IP is not available or the IP itself is not available, the connection will fail. In order to ensure load balancing and connection failover with pre-11g Release 2 clients, you will need to change the TNSNAMES.ora of the client so that it would use 3 address lines, where each address line resolves to one of the SCAN VIPs.

    

Sample TNSNAMES.ora for Oracle Database pre- 11g Release 2 Clients

sales.example.com =(DESCRIPTION=

(ADDRESS_LIST= (LOAD_BALANCE=on)(FAILOVER=ON)

(ADDRESS=(PROTOCOL=tcp)(HOST=133.22.67.192)(PORT=1521))

(ADDRESS=(PROTOCOL=tcp)(HOST=133.22.67.193)(PORT=1521))

(ADDRESS=(PROTOCOL=tcp)(HOST=133.22.67.194)(PORT=1521)))

(CONNECT_DATA=(SERVICE_NAME= salesservice.example.com)))

USING SCAN IN A MAXIMUM AVAILABILITY ARCHITECTURE ENVIRONMENT

If you have implemented a Maximum Availability Architecture (MAA) environment, in which you use Oracle RAC for both your primary and standby database (in both, your primary and standby site), which are synchronized using Oracle Data Guard, using SCAN provides a simplified TNSNAMES configuration that a client can use to connect to the database independently of whether the primary or standby database is the currently active (primary) database. In order to use this simplified configuration, Oracle Database 11g Release 2 introduces two new SQL*Net parameters that can be used on for connection strings of individual clients. The first parameter is CONNECT_TIMEOUT. It specifies the timeout duration (in seconds) for a client to establish an Oracle Net connection to an Oracle database. This parameter overrides SQLNET.OUTBOUT_CONNECT_TIMEOUT in the SQLNET.ORA. The second parameter is RETRY_COUNT and it specifies the number of times an ADDRESS_LIST is traversed before the connection attempt is terminated. Using these two parameters, both, the SCAN on the primary site and the standby site, can be used in the client connection strings. Even, if the randomly selected address points to the site that is not currently active, the timeout will allow the connection request to failover before the client has waited unreasonably long (the default timeout depending on the operating system can be as long as 10 minutes).

sales.example.com =(DESCRIPTION= (CONNECT_TIMEOUT=10)(RETRY_COUNT=3)

(ADDRESS_LIST= (LOAD_BALANCE=on)(FAILOVER=ON)

(ADDRESS=(PROTOCOL=tcp)(HOST=sales1-scan)(PORT=1521))

(ADDRESS=(PROTOCOL=tcp)(HOST=sales2-scan)(PORT=1521)))

(CONNECT_DATA=(SERVICE_NAME= salesservice.example.com)))


For more about SCAN refer:
Grid Infrastructure Single Client Access Name (SCAN) Explained (Doc ID 887522.1)

Virtual IP in RAC

How new connection establish in Oracle RAC?
For failover configuration we should need to configure our physical ip of host name in listener configuration. Listener process is accepting new connection request and handover user process to server process or dispatcher process in Oracle.
Means using listener new connection is being established by Oracle. Once connection gets established there is no need of listener process. If new connection is trying to get session in database and listener is down then what will be happening. User process gets error message and connection fails. Because listener is down in same host or something else problem. But in Oracle RAC database environment database is in sharing mode. Oracle RAC database is shared by all connected nodes. Means more than 1 listeners are running in various nodes.
In Oracle RAC database if user process is trying to get connection with some listener and found listener is down or node is down then Oracle RAC automatically transfer this request to another listener on another node. Up to Oracle 9i we use physical IP address in listener configuration. Means if requested connection gets failed then it will be diverting to another node using physical IP address of another surviving node. But during this automatically transfer, connection should need to wait up to get error message of node down or listener down using TCP/IP connection timeout. Means session should need to wait up to getting TCP/IP timeout error dictation. Once error message is received oracle RAC automatically divert this new connection request to another surviving node.
Using physical IP address there is biggest gap to get TCP/IP timeout for failover suggestion. Session should need to wait for same timeout. High availability of Oracle RAC depends on this time wasting error message.
Why VIP (Virtual IP) needs in Oracle RAC?
From Oracle 10g, virtual IP considers to configure listener. Using virtual IP we can save our TCP/IP timeout problem because Oracle notification service maintains communication between each nodes and listeners. Once ONS found any listener down or node down, it will notify another nodes and listeners with same situation. While new connection is trying to establish connection to failure node or listener, virtual IP of failure node automatically divert to surviving node and session will be establishing in another surviving node. This process doesn’t wait for TCP/IP timeout event. Due to this new connection gets faster session establishment to another surviving nodes/listener.

Characteristic of Virtual IP in Oracle RAC:
Virtual IP (VIP) is for fast connection establishment in failover dictation. Still we can use physical IP address in Oracle 10g in listener if we have no worry for failover timing. We can change default TCP/IP timeout using operating system utilities or commands and kept smaller. But taking advantage of VIP (Virtual IP address) in Oracle 10g RAC database is advisable. There is utility also provided to configure virtual IP (vip) with RAC environment called VIPCA. Default path is $ORA_CRS_HOME/bin. During installation of Oracle RAC, it is executed.
Advantage of Virtual IP deployment in Oracle RAC:
Using VIP configuration, client can be able to get connection fast even fail over of connection request to node. Because vip automatically assign to another surviving node faster and it can’t wait for TNS timeout old fashion.
Disadvantage of Virtual IP deployment in Oracle RAC:
Some more configurations is needed in system for assign virtual IP address to nodes like in /etc/hosts and others. Some misunderstanding or confusion may occur due to multiple IP assigns in same node.
Important for VIP configuration:
The VIPs should be registered in the DNS. The VIP addresses must be on the same subnet as the public host network addresses. Each Virtual IP (VIP) configured requires an unused and resolvable IP address.

Managing RAC Instances

1. Status of all RAC instaces
[oracle@naikh] srvctl status database -d testdb

2. Start all RAC instances
[oracle@naikh] srvctl start database -d testdb

3. Startup one RAC instance
[oracle@naikh] srvctl start instance -d testdb -i testdb1

4. Shutdown one RAC instance
[oracle@naikh] srvctl stop instance -d testdb -i testdb1

Rebooting RAC servers

For some maintenance activity, if RAC servers need to be rebooted then the steps are:

1. Check the status of all the services
[oracle@naikh] cd $CRS_HOME/bin
[oracle@naikh] crs_stat -t

2. Stop the database
[oracle@naikh] srvctl stop database -d orcl

3. Stop nodeapps
[oracle@naikh] srvctl stop nodeapps hostname1
[oracle@naikh] srvctl stop nodeapps hostname2

4. Reboot the servers
Ask System admin to reboot the both the servers

5. Once the server is up check the status of all the services
[oracle@naikh] cd $CRS_HOME/bin
[oracle@naikh] crs_stat -t

6. Start the nodeapps if not automatically started.
[oracle@naikh] srvctl start nodeapps hostname1
[oracle@naikh] srvctl start nodeapps hostname2

7. Start the database if not automatically started.
[oracle@naikh] srvctl start database -d orcl

8. Make sure all the services are active
[oracle@naikh] cd $CRS_HOME/bin
[oracle@naikh] crs_stat -t

ORACLE RAC Basics

Oracle Clusterware (Cluster Ready Services in 10g/ Cluster Manager in 9i) – provides infrastructure that binds multiple nodes that then operate as single server. Clusterware monitors all components like instances and listeners. There are two important components in Oracle clusterware, Voting Disk and OCR (Oracle Cluster Registry).

Voting Disk – is file that resides on shared storage and Manages cluster members.  Voting disk reassigns cluster ownership between the nodes in case of failure.

OCR (Oracle Cluster Registry) – resides on shared storage and maintains information about cluster configuration and information about cluster database. OCR contains information like which database instances run on which nodes and which services runs on which database.

CRS Resource – anything that Oracle Clusterware manages is classified as CRS resource like database, instance, service, listener, VIP address and so on.

.
Cluster-Aware Storage – is storage solution for Oracle RAC like RAW device, OCFS, ASM
.
Interconnect – is private network that connects all the servers in cluster. Interconnect uses switch that only nodes in cluster can access. Instances in cluster communicate to each other via interconnect.
.

Cache Fusion – is disk less cache coherency mechanism in Oracle RAC that provides copies of data blocks directly from one instance’s memory cache (in which that block is available) to other instance (instance which is request for specific data block).  Cache Fusion provides single buffer cache (for all instances in cluster) through interconnect.

In Single Node oracle database, an instance looking for data block first checks in cache, if block is not in cache then goes to disk to pull block from disk to cache and return block to client.

In RAC Database there is remote cache so instance should look not only in local cache (cache local to instance) but on remote cache (cache on remote instance). If cache is available in local cache then it should return data block from local cache; if data block is not in local cache, instead of going to disk it should first go to remote cache (remote instance) to check if block is available in local cache (via interconnect)

This is because accessing data block from remote cache is faster than accessing it from disk.

.

Cache Fusion Model
Cache fusion Model is dependent on three services
– Global Resource Directory (GRD)
– Global Cache Service (GCS)
– Global En-queue Service (GES) and –
.
SSH User Equivalency – means assigning same properties (username, userid, group, group id and same password) to operating system user (installing & owning RAC database) across all nodes in cluster

CVU (Cluster Varification Utility) – is utility to verify that system meets all the criteria for Oracle Clusterware Installation.

 

Storage Options for RAC

  • CFS (Cluster File System) – Easy to manage but only available on some platforms.  Does not address striping and mirroring.
  • RAW – Available on all platforms but difficult to manage. Does not address striping and mirroring.
  • NFS – Easy to manage but only available on some platforms.  Does not address striping and mirroring.
  • ASM (Automatic Storage Management) – Easy to manage, available on ALL platforms, and DOES address striping and mirroring.

CFS (Cluster Filesystems)

  • The idea of CFS is to basically share file filesystems between nodes.
  • Easy to manage since you are dealing with regular files.
  • CFS is configured on shared storage.  Each node must have access to the storage in which the CFS is mounted.
  • NOT available on all platforms.  Supported CFS solutions currently:
  • OCFS on Linux and Windows (Oracle)
  • DBE/AC CFS (Veritas)
  • GPFS (AIX)
  • Tru64 CFS (HP Tru64)
  • Solaris QFS

RAW (character devices)

  • Hard to manage since you are dealing with character devices and not regular files.
  • Adding and resizing datafiles is not trivial.
  • On some operating systems volume groups need to deactivated before LVs can be manipulated or added.

NFS (Network Filesystem)

  • NOT available on all platforms.  Supported NFS solutions currently:
  • Network Appliance
  • Redhat Linux
  • Fujitsu Primecluster
  • Solaris Suncluster

ASM

  • Stripes files rather than logical volumes.
  • Enables online disk reconfiguration and dynamic rebalancing.
  • Provides adjustable re balancing speed.
  • Provides redundancy on a file basis.
  • Supports only Oracle files.
  • Is cluster aware.
  • Is automatically installed as part of the base code set .

Source : Internet

SCAN -Single Client Access Name

rac-scan2

Figure 1

SCAN is new Oracle Real Application Clusters (RAC) 11gR2 feature. It gives a single name for clients to access oracle database running in a cluster.
Basics of SCAN

– Single client access name (SCAN) is the virtual hostname to provide for all clients connecting to the cluster (as opposed to the vip hostnames in 10g and 11gR1).

– SCAN is a domain name registered to at least one and up to three IP addresses, either in the domain name service (DNS) or the Grid Naming Service (GNS).

– By default, the name used as the SCAN is also the name of the cluster and must be globally unique throughout your enterprise.

– For installation to succeed, the SCAN must resolve to at least one address. There can be a maximum of three ip addresses that can be used for the SCAN.

– SCAN VIP addresses must be on the same subnet as virtual IP addresses and public IP addresses.

– SCAN VIPs can float in the cluster while Node VIPs as default run on specific nodes only.

– SCAN Listeners are simply used for redirecting the request received from client, this is usually considered a lightweight process when we compare it to the functioning of Node VIP Listeners as these actually fork a new process to create database conenction.

–  SCAN IP address and SCAN listeners are managed as resources in Oracle Clusterware.

– For high availability and scalability, Oracle recommends that you configure the SCAN to use DNS Round Robin resolution to three addresses.

Benefits of SCAN

1) the SCAN makes it possible to add or remove nodes from the cluster without needing to reconfigure clients (Because the SCAN is associated with the cluster as a whole, rather than to a particular node). Before SCAN, if the cluster changed, then the client TNSNAMES.ORA files (or other tns connect strings like Easy Connect strings) would need to change. So your DBA  can easily add or delete a node without changing the configuration files.

For example, connection to database “11g” using SCAN would use this tnsnames entry:

11g =
 (DESCRIPTION=
 (ADDRESS=(PROTOCOL=tcp)(HOST=rac-scan.corp.to)(PORT=1521))
 (CONNECT_DATA=(SERVICE_NAME=11g))
 )

While without SCAN we were using

11g_old =
 (DESCRIPTION=
 (ADDRESS_LIST=
 (ADDRESS=(PROTOCOL=tcp)(HOST=rac1-vip.corp.to)(PORT=1521))
 (ADDRESS=(PROTOCOL=tcp)(HOST=rac2-vip.corp.to)(PORT=1521))
 )
 (CONNECT_DATA=(SERVICE_NAME=11g))

2) It also adds location independence for the databases, so that client configuration does not have to depend on which nodes are running a particular database.

Working

“When a client submits a request, the SCAN listener listening on a SCAN IP address and the SCAN port is contracted on a client’s behalf. Because all services on the cluster are registered with the SCAN listener, the SCAN listener replies with the address of the local listener on the least-loaded node (Each scan listener keeps updated cluster load statistics) where the service is currently being offered. Finally, the client establishes connection to the service through the listener on the node where service is offered.All of these actions take place transparently to the client without any explicit configuration required in the client.”

Scan_oracle_rac_image

Figure 2 ( taken from Oracle documentation)

Figure 2 above shows where SCAN and Local listeners fits into picture.

Connection flow in case of SCAN is :

1) Client sends connection to SCAN name and a SCAN IP address is provided in return if DNS is used.  SCAN IPs are returned in round-robin process. If GNS is used for SCAN ip management then DNS delegates the request to GNS services and GNS services in turn return a SCAN IP address ( again in round-robin fashion)

2) The TNS request is now forwarded to the SCAN Listeners (shown in Fig 2)

3) SCAN listeners in turn forward the request to local listeners ( least loaded one)

4) Local listeners take care of client request.

Setup

In order to successful install grid infrastructure you need to configure your DNS  prior to  installing the grid infrastructure to resolve the name accordingly. Oracle requires at least one IPs to be configured for the scan name.

You can have two options to define SCAN name

1) Define it in your DNS (Domain Name Service

2) Use GNS (Grid Naming Service)

Also after the setup is done, the remote_listener parameter is made to point to SCAN  tnsnames.ora entry and local_listener uses VIP Listener entry. The  remote listeners which points to SCAN listeners will do the load balancing and local listeners will take care of new process spawning and connection to database. PMON registers with SCAN listener as defined in parameter ‘ remote_listener’ setting and also with the node listener depending on the local listener settings. On the basis of  PMON provided details SCAN will choose the least loaded node to forward the request received from client.

NAME                                    TYPE         VALUE
———————————— ———– ——————————
remote_listener                        string         rac-scan.global.to:1521
NAME                          TYPE          VALUE
———————————— ———– ————————————————
local_listener         string         (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP) (HOST=196.1.2.50)(PORT=1521))))

You need to provide SCAN related settings during installation of Grid infrastructure as shown below.

scan_install_screen

Figure 3

Is it mandatory to use SCAN?

No but It is highly recommended to use SCAN unless there’s strong business reason preventing it from being used.

RAC 11g Useful Commands

Some useful RAC 11G commands are documented below.

 STOP SEQUENCE

START SEQUENCE

OTHER USEFUL COMMANDS