Data Guard Broker

The broker is part of Oracle's Database Enterprise Manager and an integral part of Data Guard, it is the management framework. You can of course use SQL*Plus to manage Data Guard but the broker when used with the Enterprise Manager becomes a powerful tool offering the following

• Configuration
• Monitoring
• Alerting
• Performance Analysis
• Manual switchover's
• Automatic switchover's

Before I discuss the integration of the broker into the Enterprise Manager lets first discuss the broker itself. The broker can make changes to a Data Guard setup (physical or logical database), the type of changes are configuration, transport modes, apply setup and role change services and the protection mode. You can also monitor the health of any configuration, it is also responsible for implementing and managing the automatic failover capability also know as Fast-Start Failover (FSFO).

The broker has three parts, a set of background processes on each database, a set of configuration files and a commandline interface (CLI) called DGMGRL.

The broker has a number of processes running, the broker manages these processes automatically you have no control over them

• Data Guard Monitor (DMON) - this is the main process and is responsible for coordinating all broker actions as well as maintaining the configuration files, this process is enabled or disabled with the DG_BROKER_START parameter
• Broker Resource Manager (RSM) - is responsible for handling any SQL commands used by the broker that need to be executed on one of the databases in the configuration. These SQL commands are made as a result of a change to the configuration made through DGMGRL or are the configuration commands executed by the broker during the database startup
• Data Guard Net Server (NSVn) - from 1 to n can exist, they are responsible for making contact with the remote databases and sending across any work items to the remote database.
• DRCn - these network receiver processes establish the connection from the source database NSVn process, this is a similar connection to the LogWriter Network Service (LNS) to a Remote File Server (RFS) connection for redo transport. When the broker needs to send something (data or SQL) between databases it uses this NSV to DRC connection, these connection are started as needed.
• Configuration files - the configuration is stored in two binary command files. The files contain the complete configuration setup, the database states, how to connect to each one and what parameters to setup when each database starts up.

In a broker configuration it is the Data Guard Monitor (DMON) process on the primary database that is the owner of the configuration, all orders will come from the primary even if use a DGMGRL CLI from another server, the standby database will receive all configuration changes via the primary.

When the DMON process communicates with the standby database it uses one of the NSV processes to send the work to the standby, this protects the DMON from a hang if the network should go down. Whenever the DMON needs to execute some SQL it will enlist the aid of the RSM process on the primary database, if the SQL is for the primary database it will execute it directly, however if the SQL is for a standby database the RSM process asks the NSV process to send it to the standby database again this protects the RSM from a network hang.

Each NSV process has a DRC partner process on the standby database, which will perform the work of the NSV process, the DRC process on the standby will return the results or status back to the NSV on the primary.

On startup the DMON will communicate with each standby via the NSV-DRC connection process pair to establish communication and configuration information so that the standby can start the apply services, if there are any problems check the alert.log after the NVS section to see if the are any TNS errors.

When using a RAC environment there are another set of process called the internode servers (INSV) which maintain a connection between the nodes in the cluster to ensure that the broker on each node knows the state of the cluster. A INSV process will always be started on the primary even if the primary is not in a RAC.

Lastly you may see one more process called the Fast-Start-Failover Process (FSFP), which is used only when the primary database is under the control of Data Guard automatic failover feature Fast-Start Failover. We will be discussing FSFP in another section switchover and failover.

Configuration Files

Each database has two copies of the configuration, stored in the below parameter locations

• DG_BROKER_CONFIG_FILE1
• DG_BROKER_CONFIG_FILE2

By default they are stored in $ORACLE_HOME/dbs directory with the filename of dr1<DB_UNIQUE_NAME>.dat and dr2<DB_UNIQUE_NAME>.dat, if using a RAC these should be keep on a shared location as only one set of copies can be exist for the entire RAC.

The primary has the master copies of the configuration files, all changes to these files are done by the primary, even if you are on the standby server the configuration changes will be passed to the primary via NSV-DRC processes and the DMON process will make the change which is then propagated back to the standby databases. The reason for the multiple copies of the configuration file is if the primary has a problem then each standby database has a copy, when a failover occurs the standby that becomes the primary will then become the master of those configuration files.

Broker

You have two choices to interact with Data Guard either Enterprise Manager (EM) or the broker CLI DGMGRL you can swap between the two with a few simple configuration changes. To gain full full functionality of Data Guard through Grid Control you must use the broker.

Broker CLI DGMGRL comes with Enterprise Manager, you can run the broker on any platform you wish it does not need to be the same platform as the primary or standby databases.

You can access the DGMGRL command line using the below

When you start the DGMGRL commandline it does not connect you to a database, only the Data Guard configuration. Before you start to use the broker there are a number of things to perform first

• Configure the Broker parameters
• The broker and the listener
• RAC and the broker
• Connecting to the broker

Even if you use Enterprise Manager you should really have a understanding on how the CLI works, as mentioned above there are two configuration files, both the systems parameters must be set on the primary and all standby databases, you put set this files anywhere you like

In a RAC environment they should be in a shared area and all nodes should point to the same files, only one set of files must exist in a RAC.

Once the directory has been created you are ready to startup the broker, do this on all databases (primary and standby)

When you start the broker it does not configure any files yet because you have to use the DGMGRL command, all it does is start the necessary processes we mentioned earlier. If you need to change the location of the files the steps are below

1. stop the broker using the DG_BROKER_START parameter by setting this to false (on all nodes in the cluster)
2. change the DG file destination parameters
3. copy the files to the new location
4. then re-start the broker, again set the DG_BROKER_START parameter to true (on all nodes in the cluster)

The broker uses the Oracle Net Services to make connections to the databases, setup both redo transport and archive gap resolution and perform role transitions. We need to create a special static entry in the listener.ora file for each database in the broker configuration, this entry makes it possible for the broker to connect to an idle instance using a remote SYSDBA connection and perform the necessary startup. Here is an example

Once the above has been done we are ready to create a broker configuration, there is no difference in a non-RAC or RAC environment, to connect to the configuration you use the dgmgrl command but one word of advise make sure that you use a specific user and password and not the / as the user, this has know to cause problems at a later date

There are four main area's to DGMGRL CLI

• Connection and Help - connect, help and exit
• Creation and Editing - create, add, enable, edit and convert
• Monitoring - show
• Role Transition - switchover, failover and reinstate

If you used the Enterprise Manager/Grid Control to setup your standby database then the next steps would have already been performed for you, if however you used the Power User Method then you need to follow the next steps to configure the broker.

The first step is to create the base configuration

The above command would have created a base configuration, you can display the configuration using the below command

Next we add the standby database to the configuration, if you receive an error stating that the properties could not be imported from the database, you need to check DB_UNIQUE_NAME also the TNSNAME and the transport parameter (LOG_ARCHIVE_DEST_n), so what is the proper setup, you must have your redo transport parameter defined using the DB_UNIQUE_NAME method, meaning that each redo transport parameter must contain the DB_UNIQUE_NAME=<name> attribute. The broker will search all of your LOG_ARCHIVE_DEST_n parameters looking for a database with a unique name that matches the database name you entered for the command. If you have not done this then you need to use the full set of arguments to add the database

When you display the configuration you should see that the standby database has been added

One parameter to watch for is the local archiving parameter on the primary and standby databases, the broker will modify this parameter and if necessary add the VALID_FOR attribute in preparation for the archival of your standby redo log files.

One area to watch is if you add an additional standby database, the broker could presume the wrong parameter values, for instance you may want one standby to use SYNC and another to use ASYNC, but if you configure the second standby database using the short method it may default to SYNC, so when you do add any additional standby databases always double check the parameters are set correctly for the type of configuration that you want, any changes should be done before enabling the additional standby database.

Once we are happy that all standby databases have been added and their parameters have been check we are ready to enable the configuration, the command will issue alter system commands on both the primary and standby databases, start the redo transport to the standby databases and the apply services, before examining the configuration give it sometime to complete as it does take a while, otherwise you see some ORA-16610 errors.

You can tail the alert log file and see what action is taking place, hopefully after a few minutes you should see a successful configuration running, you can also check the redo transport and apply services by viewing the v$managed_standby view.

Broker Properties

As I mentioned above the broker has a number of properties which can be changed, to change the property we use the EDIT command in the DGMGRL CLI

Broker State and Monitoring

There are a number of commands that you can use to change the state of the database

There are a number of useful monitoring commands and log files that can help with diagnosing problems

Lastly there is a utility called the "DG Menu Utility", this can be downloaded from the Oracle Data Guard 11g handbook web site http://www.dataguardbook.com, I will be covering this in my monitoring section but below is a screen shot of the menu

Removing the Broker

Finally if you wish remove the broker follow the steps below