The Oracle Exadata plug-in for OEM enables us to monitor as well as manage Exadata targets through Enterprise Manager Cloud Control 12c.
The features include:
- Monitoring of the Exadata Database Machine as an Enterprise Manager target.
- Monitoring of the all the components of the Exadata Database Machine target, including the Exadata Storage Cells
- Execute CELLCLI commands and also configure and manage I/O Resource Management (IORM) feature.
- Support SNMP notification for Exadata cells.
- Support dashboard report creation from Enterprise Manager Cloud Control for Exadata Database Machine
- Enhanced InfiniBand network fault detection and InfiniBand schematic port state reporting.
This note describes the procedure used to deploy the plug-in on an Exadata X5-2 one eighth rack.
The version of the Oracle Enterprise Manager agent is 12.1.0.5 and the Exadata plug-in version is 12.1.0.6.
At a high level, the Exadata plug-in installation and deployment consists of the following steps:
- Create a database server ILOM Service Processor user
- Verify ILOM Ipmitool version
- Verify Infiniband Switch firmware
- Obtain the credentials needed for Agent installation and Target Discovery
- Install Enterprise Manager Agents on compute nodes
- Setup SSH equivalence between the cellmonitor user and the agent (oraagent) user
- Discover the Exadata Database Machine
- Post-Discovery Configuration and Verification
- Create Exadata Database Machine Dashboard
Create a Database Server ILOM Service Processor User
- Log in to the database server ILOM Service Processor (exdb1dbadm01-ilom) as root
login as: root
Using keyboard-interactive authentication.
Password:
Oracle(R) Integrated Lights Out Manager
Version 3.2.4.52 r101649
Copyright (c) 2015, Oracle and/or its affiliates. All rights reserved.
Hostname: exdb1dbadm01-ilom
- Change to the users directory
-> cd /SP/users
/SP/users
- Create the oemuser user and password
-> create oemuser
Creating user…
Enter new password: ********
Enter new password again: ********
Created /SP/users/oemuser
- Change to the new user’s directory and set the role
-> cd oemuser
/SP/users/oemuser
-> set role=’cro’
Set ‘role’ to ‘cro’
Note: Repeat all the same steps for exdb1dbadm02-ilom
- Test the ILOM user ID oemuser created above, by listing the last 10 system events:
Run the command below on both compute nodes:
ipmitool -I lanplus -H exdb1dbadm01-ilom -U oemuser -P welcome1 -L USER sel list last 10
[root@exdb1db01 ~]# ipmitool -I lanplus -H exdb1dbadm01-ilom -U oemuser -P welcome1 -L USER sel list last 10
35a | 01/18/2016 | 05:19:22 | System Firmware Progress | Hard-disk initialization | Asserted
35b | 01/18/2016 | 05:19:22 | System Firmware Progress | Option ROM initialization | Asserted
35c | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted
35d | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted
35e | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted
35f | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted
360 | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted
361 | 01/18/2016 | 05:20:14 | System Firmware Progress | Option ROM initialization | Asserted
362 | 01/18/2016 | 05:20:22 | System Firmware Progress | System boot initiated | Asserted
363 | 01/18/2016 | 05:20:22 | System Firmware Progress | System boot initiated | Asserted
[root@exdb1db02 ~]# ipmitool -I lanplus -H exdb1dbadm01-ilom -U oemuser -P welcome1 -L USER sel list last 10
35a | 01/18/2016 | 05:19:22 | System Firmware Progress | Hard-disk initialization | Asserted
35b | 01/18/2016 | 05:19:22 | System Firmware Progress | Option ROM initializat ion | Asserted
35c | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted
35d | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted
35e | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted
35f | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted
360 | 01/18/2016 | 05:20:14 | System Firmware Progress | Hard-disk initialization | Asserted
361 | 01/18/2016 | 05:20:14 | System Firmware Progress | Option ROM initialization | Asserted
362 | 01/18/2016 | 05:20:22 | System Firmware Progress | System boot initiated| Asserted
- 01/18/2016 | 05:20:22 | System Firmware Progress | System boot initiated| Asserted
Verify ILOM Ipmitool version
Enterprise Manager requires a minimal Ipmitool software version of 1.8.10.3 for Oracle Linux.
Using DCLI, we can check the version on both compute nodes by running the ipmitool –V command.
[root@exdb1db01 ~]# dcli -g ~/dbs_group -l root ipmitool -V
exdb1db01: ipmitool version 1.8.12.3
exdb1db02: ipmitool version 1.8.12.3
The dbs_groupfile contains the list of compute node hostnames, one on each line.
Create the dbs_groupfile if it does not exist.
[root@exdb1db01 ~]# cat dbs_group
exdb1db01
exdb1db02
Verify Infiniband Switch firmware
Log on to the management interface for the InfiniBandSwitch (exdb1sw-iba01) as root.
login as: root
root@exda1sw-iba01’s password:
Run the command nm2version.
The minimum version should be 1.1.3-2
[root@exda1sw-iba01 ~]# nm2version
(Note: The command ‘nm2version’ is deprecated and will be removed from future
releases. Please use ‘version’ instead)
SUN DCS 36p version: 2.1.5-1
Build time: Oct 6 2014 10:35:15
SP board info:
Manufacturing Date: 2015.02.17
Serial Number: “NCDIV0061”
Hardware Revision: 0x0200
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010
Note: Perform the same check on the second Infiniband Switch.
Obtain the credentials needed for Agent installation and Target Discovery
List of credentials needed for Discovery
- Agent host credentials: the username and password of the user running the agents on the compute nodes
Note – Agent owner should be in the same OS group as the CRS and Database software owners
- Monitoring credentials:
- Storage Cell Root password
- nm2user password of each InfiniBand switch
- ILOM username and password
- InfiniBand Switch root credentials: Username and Password
Install Enterprise Manager Agents on compute nodes
From the Setup menu select Add Target followed by Add Targets Manually
Select option “Add Host Targets”
Click Add Host ….
Click Add
Enter the names of both the compute nodes as well as the directory path location on the compute nodes for installing the 12.1.0.5 Agent software.
The agent software base directory is /opt/oraagent
Note – In this example we are adding one compute node at a time, but we can do the agent installation on both the compute nodes in one step.
Provide the credentials of the OS user we are going to use for the agent deployment (oraagent)
Review and click on Deploy Agent
Note: After the agent deployment is complete, execute the root.sh script on each compute node
After the agent has been installed and deployed on both compute nodes, we should be able to see both agents up and running as shown below:
Setup SSH equivalence between the cellmonitor user and the agent (oraagent) user
On each of the compute nodes we will run the command ssh-keygen –t dsa as the agent user oraagent to generate the agent’s SSH public key.
Then copy that file to each of the Storage Cells and add the contents of the file id_dsa.pub to the authorized_keys file belonging to the cellmonitor user.
Basically the oraagent user needs to be able to connect to each of the Storage Cells as the cellmonitor user and execute the command “cellcli –e list cell attributes msstatus”.
Otherwise after the Exadata Database Machine discovery is completed, in the schematic diagram we will see the Storage Cells displaying with a red colour down status when actually all the CELLSRV processes are running.
Verify if the cellmonitor user can SSH to one of the Storage Cells and no password is being prompted
Discover the Exadata Database Machine
Select Add Targets Manually from the Setup and Add Target menu
Select the option “Add Targets Using Guided Process” and in the Target Types drop-down list select “Oracle Exadata Database Machine”
Click on the button Add Using Guided Process
Select the option to Discover a new Database Machine and its hardware components and click on Discover Targets
In the Discovery Inputs screen specify the agent that you would want to use for discovery. You can choose any one of the compute nodes but it is recommended to use the first node of the cluster.
Provide the credentials for the agent.
Click on Set Credential and enter the details of the agent user account on the compute node
Select the schematic file as databasemachine.xml and check the File Location corresponds to the /opt/oracle.SupportTools/onecommand directory
For the Infiniband network discovery, we create a new credential. The credentials of the nm2user are used here.
A prerequisite check is now performed to verify that the databasemachine.xml file is present and is readable by the OEM agent.
Verify the names and IP addresses of the Database Machine components which have been discovered.
For each of the Database Machine components, a monitoring agent is assigned as well as a backup agent. We can either use the in-built random assignment or manually assign primary monitoring and backup agents for each component.
We now provide the credentials of the OEM agents running on each of the compute nodes. Test the connection as well.
In the Monitoring Credentials screen, we need to provide the root passwords for the Storage Servers (Cells), nm2user for the Infiniband Switches, and the oemuser for the ILOMs.
This screen is for the configuration of SNMP subscriptions for the OEM agents to receive SNMP traps.
For the SNMP Community String provide the value ‘Public’ and provide the root passwords for both the Infiniband Switches.
Review and then click on Submit.
Then review the summary of the Target Creation .
Ensure each component has a successful Discovery Status.
After the Database Machine discovery is complete, we will see a new menu option called “Exadata” added to the Targets menu in OEM 12c Cloud Control
Launch DB Machine Home
Click on the Exadata target and that will take us to the Oracle Exadata Schematic page
Post-Discovery Configuration and Verification
- Set Up SNMP Trap Forwarding on Compute Node
We need to set up SNMP trap forwarding on the compute node where the Enterprise Manager Agent is running so that the snmptrapd daemon can receive traps using port 162 and forward the trap to the port the OEM agent is listening on.
In this case we have verified from the emd.properties (property EMD_URL) file located in the $AGENT_INST/sysman/config directory that the value of the port is 3872
Confirm that the agent receivelet is listening on this port over UDP
Then stop the SNMP trap daemon (in this case it is already stoped – otherwise issue command service snmptrapd stop)
Add the following entry to the /etc/snmp/snmptrapd.conf configuration file:
authcommunity log,execute,net <community string>
forward default udp:localhost:3872
Note: Do this on both compute nodes
Edit the /etc/hosts.allow file to add the following line:
snmptrapd : ALL
Start the SNMP trap daemon and ensure via chkconfig command that the daemon will start on a node reboot
- Verify SNMP Configuration for Storage Cells
For Exadata plug-in Release 12.1.0.2.0 and later, SNMP trap setup for Exadata Storage Cells is done automatically during the discovery process
Verify
[root@exdb1celadm01 ~]# cellcli -e list cell attributes snmpSubscriber
((host=exdb1dbadm02.rac.com.au,port=1830,community=public),(host=exdb1dbadm01.rac.com.au,port=1830,community=public),(host=exdb1db01.rac.com.au,port=3872,community=public),(host=exdb1db02.rac.com.au,port=3872,community=public),(host=61.88.37.93,port=162,community=public,type=ASR))
[root@exdb1celadm01 ~]# cellcli -e list cell attributes notificationMethod
mail,snmp
[root@exdb1celadm01 ~]# cellcli -e list cell attributes notificationPolicy
critical,warning,clear
- Configure and Verify SNMP for InfiniBand Switch Targets
For Exadata plug-in Release 12.1.0.2.0 and later during the discovery process, we provide the necessary root credentials to set up SNMP trap for the InfiniBand Switch.
SNMP trap setup for Infiniband Switches is done automatically during the discovery process.
We can verify this by connecting via ILOM web interface to the Infiniband Switch.
Click Configuration, then System Management Access, and finally SNMP
Ensure the following values are set:
State=Enabled
Port=161
Protocols=v1,v2c,v3
Verify the InfiniBand Switch SNMP configuration for Enterprise Manager monitoring
From one of the compute nodes execute the command:
snmpget -v 1 -c public exdb1sw-iba01 1.3.6.1.4.1.42.2.70.101.1.1.9.1.1.5
[root@exdb1db01 onecommand]# snmpget -v 1 -c public exdb1sw-iba01 1.3.6.1.4.1.42.2.70.101.1.1.9.1.1.5
SNMPv2-SMI::enterprises.42.2.70.101.1.1.9.1.1.5 = INTEGER: 1
Note: We should not get a Timeout message. If so, it means that the SNMP configuration for OEM monitoring is not complete.
Create Exadata Database Machine Dashboard
We create the Database Machine dashboard to give us a single pane-of-glass view of the performance and usage metrics of the Exadata Database Machine system as well as its components.
Navigate to “Enterprise” -> “Job” -> “Activity”
