Saturday, August 31, 2013
Adventures in Hadoop: #1 The First Step is the Most Important
I consider myself a tenaciously curious person. In the spirit of "discovery" I've embarked on learning Hadoop and, subsequently the various bits and pieces that are associated with it. Since I am now "into" blogging, it occurred to me that there might be others like myself who are keen on learning about Hadoop. Below is a listed of useful blogs I visited in my quest of knowledge.
This picture (from my daughters room) aptly represents me vs BIG DATA :)
Note: This blog is a Work In-Progress (WIP). Please revisit it frequently for updated content :)
Hadoop
Without a doubt a useful technology when applied to the correct use-case. I think it all boils down to "What is your question?". But, before I got too philosophical, the more relevant question was "How does it work?". I stumbled on to Michael Noll's tutorial on configuring a Single Node Cluster. He did an amazing job creating step by step documentation on the setup. It was easy enough to configure it and test with the Gutenberg examples
Don't forget to check out the Web Interface for the NameNode, JobTracker and, TaskTracker.
Pig
At this point, I was thinking "Great, I have a Hadoop install but, how do I easily get it to do my work?". I mean, I can program in Java but I'm no Ace! Enter, Pig Latin.
Once again, I found an excellent article by Wayne Adams which outlined how to leverage Pig to "ask" the question. He used the data dumps available for New Issues Pool Statistics to illustrate how Pig Latin is utilized on Hadoop.
Hive
Again, as I mentioned above, I'm from a DBA background so queries are familiar to me. Hive is a great add-on to Hadoop which allows for a SQL interface approach to NoSQL. I'm thinking External Tables in Oracle when I created the tables from Ben Hidalgo's example.
Conclusion
I tend to drift towards over-simplification at times, and since I come from a DBA background with development roots, I like to use the "You get what you ask" analogy when dealing with an instance. For example, if you ask for a lot of data, well, you're going to get it and - unless you're on something like an Exadata machine - it might take a while. You know, as a stupid question and you'll get a stupid answer type of deal. The point of my rant is, from what I surmise, Hadoop (NoSQL) has its place for certain use-cases and the "right" solution depends on the "right" question.
I'm planning to rebuild this environment because its been a couple of weeks since I last tinkered with it. I aim to provide more details on this blog for each step. I've started working with R and how - at the very least I - can use it for my every day work.
Next in Series: #2 Starting from Scratch
Next in Series: #2 Starting from Scratch
Other Useful Links
Thursday, August 15, 2013
A Moment of Honesty - So refreshing
I was chatting with one of our interns this morning about an customers issue. The topic was something to do with removing a target from OEM using EMCLI and, as usual, I wanted it to be a good learning experience for the intern. Though, at one point after the issue was resolved, the intern was compelled to say this to me:
Cheers!
INTERN> a moment of honesty and do not take this the wrong way at all. you're a scary teacher but your method makes you a very good one lol
INTERN> one of the best i have ever had
INTERN> i only mean scary in the sense that you ask us a lot of questions to pull out answers and not knowing the answer can be unsettling in my mind despite me still learning hahaI couldn't help but chuckle and laugh at this - the intern is a good kid and quite good at picking up whatever I throw at him - and, I guess I tend to expect much. I suppose both of us learned something today: I, told the intern that I will try not to be as scary in the future and the intern, that I am not as scary as I seem to be :)
Cheers!
Wednesday, August 7, 2013
Oracle Enterprise Manager 12c Windows Agent Deployments Made Easy
As of late, I've had to do a few Windows OS Agent Deployments. As we all know, if you want to auto deploy to a Windows OS, you need to have CygWin installed (on each target host) first.
The alternative, via Silent Install, is illustrated by my good friend Bobby Curtis here. However, for large scale quicker deployments, I came up with an a solution which worked quite well.
Assuming you have already download the agent binaries from the OMS server, copy them to a centralized location accessible to all target hosts. In my case, I used \\slave\software\oracle\oem\12.1.0.3.0\agent
The important thing here is the content of the agent.rsp file. Mine looks like this. I did not keep the ORACLE_HOSTNAME parameter within the response file (agent.rsp)
The alternative, via Silent Install, is illustrated by my good friend Bobby Curtis here. However, for large scale quicker deployments, I came up with an a solution which worked quite well.
Assuming you have already download the agent binaries from the OMS server, copy them to a centralized location accessible to all target hosts. In my case, I used \\slave\software\oracle\oem\12.1.0.3.0\agent
The important thing here is the content of the agent.rsp file. Mine looks like this. I did not keep the ORACLE_HOSTNAME parameter within the response file (agent.rsp)
####################################################################
## copyright (c) 1999, 2012 Oracle. All rights reserved. ##
## ##
## Specify values for the variables listed below to customize ##
## your installation. ##
## ##
## Each variable is associated with a comment. The comment ##
## identifies the variable type. ##
## ##
## Please specify the values in the following format: ##
## ##
## Type Example ##
## String "Sample Value" ##
## Boolean True or False ##
## Number 1000 ##
## StringList {"String value 1","String Value 2"} ##
## ##
## The values that are given as need to be ##
## specified for a silent installation to be successful. ##
## ##
## ##
## This response file is generated by Oracle Software ##
## Packager. ##
###################################################################
RESPONSEFILE_VERSION=2.2.1.0.0
#-------------------------------------------------------------------------------
#OMS_HOST: OMS host info required to connect to OMS
#EM_UPLOAD_PORT: OMS port info required to connect to OMS
#AGENT_REGISTRATION_PASSWORD: Agent Registration Password needed to
# establish a secure connection to the OMS.
#AGENT_INSTANCE_HOME: Agent instance home is the location of agent state directory.
#AGENT_PORT: Agent port on which the agent process should be started.
#b_startAgent: Agent will not be started after configuration if the value specified is false.
#ORACLE_HOSTNAME: Fully qualified domain name of host where is the agent is deployed.
#s_agentHomeName:Customized Oracle home name for the agent home. Example: s_agentHomeName="agent12gR1"
#-------------------------------------------------------------------------------
OMS_HOST=cruiser.missile.com
EM_UPLOAD_PORT=4899
AGENT_REGISTRATION_PASSWORD=you_havent_changed_the_sysman_password
AGENT_INSTANCE_HOME=D:\oracle\app\product\12.1\agent
AGENT_PORT=3875
b_startAgent=true
s_agentHomeName=agent12c_home1
#-------------------------------------------------------------------------------
#s_agentServiceName: Sets the agent Service Name and this variable can be
# used to overrite the agent service name calculated by the install. This is
# required for only Windows.
# Example:
# s_agentServiceName = "Oracleagent12gAgent" ; default value
# s_agentServiceName = "GridAgent" ; User specified value
#-------------------------------------------------------------------------------
s_agentServiceName="Oracleagent12cAgent"
####################################################################################
#Please Don't change the values of these variables
####################################################################################
#-------------------------------------------------------------------------------
#EM_INSTALL_TYPE: install type
#-------------------------------------------------------------------------------
EM_INSTALL_TYPE="AGENT"
I then created a wrapper install script. Since I was deploying on windows, the script looked like
c:\> view DeployTheAgent.bat \\slave\software\oracle\oem\12.1.0.3.0\agent\agentDeploy.bat AGENT_BASE_DIR=D:\Oracle\app\product\12.1 RESPONSE_FILE=\\slave\software\oracle\oem\12.1.0.3.0\agent\agent.rsp ORACLE_HOSTNAME=%COMPUTERNAME%.%USERDNSDOMAIN%
The linux version of this might look something like
$ cat DeployTheAgent.sh \\slave\software\oracle\oem\12.1.0.3.0\agent\agentDeploy.sh AGENT_BASE_DIR=/u01/app/oracle/product/12.1 RESPONSE_FILE=<path to mounted drive>\<agent software directory>\agent.rsp ORACLE_HOSTNAME=hostname.domainname
From here, its a simple as invoking the script from the target host:
c:\>\\slave\software\oracle\oem\12.1.0.3.0\agent\DeployTheAgent.bat \\slave\software\oracle\oem\12.1.0.3.0\agent\agentDeploy.bat AGENT_BASE_DIR=D:\Oracle\app\product\12.1 RESPONSE_FILE=\\slave\software\oracle\oem\12.1.0.3.0\agent\agent.rsp ORACLE_HOSTNAME=javelin.missile.com \\slave\software\oracle\oem\12.1.0.3.0\agent Present working directory:\\slave\software\oracle\oem\12.1.0.3.0\agent Archive location:\\slave\software\oracle\oem\12.1.0.3.0\agent directory AGENT_BASE_DIR AGENT_BASE_DIR D:\Oracle\app\product\12.1 Agent base directory:D:\Oracle\app\product\12.1 D:\Oracle\app\product\12.1 RESPONSE_FILE \\slave\software\oracle\oem\12.1.0.3.0\agent\agent.rspORACLE_HOSTNAME javelin.missile.com Agent base directory:D:\Oracle\app\product\12.1 OMS Host: Agent image loc : "\\slave\software\oracle\oem\12.1.0.3.0\agent" D:\Oracle\app\product\12.1 configonlyfalse Reading the properties file: "\\slave\software\oracle\oem\12.1.0.3.0\agent"\agentimage.properties 1 file(s) copied. This is the version 12.1.0.2.0 This is the type core This is the aru id 233
The installer takes a few minutes and I'd rather not bore you with the contents of the log files. You know what you are doing :)
So, for any new target host, simply invoke the DeployAgent.bat or DeployAgent.sh script and relax! Assuming that your paths are correct and, network ports are accessible, the installation should go through without any glitches.
Reference:
Oracle Documentation: Installing Oracle Management Agent in Silent Mode
So, for any new target host, simply invoke the DeployAgent.bat or DeployAgent.sh script and relax! Assuming that your paths are correct and, network ports are accessible, the installation should go through without any glitches.
Reference:
Oracle Documentation: Installing Oracle Management Agent in Silent Mode
Tuesday, August 6, 2013
Copy Paste with Command Prompt - Save your "Clicks"!!
Okay, so this post has nothing to do with Oracle but everything to do with my hesitance for having to use Command Prompt utility in windows. Even though I like hitting the "up" arrow to recall previous commands, Copy and Pasting is a pain!
So, in the spirit of sharing and in my utter innocence and assumption that most of my readers are as clueless as I am, I'd like to share something new I learned about CMD.EXE :) A certain new friend of mine, whose name I will not mention except that his initials are NC, enlightened me today.
Often enough, we need to copy and paste content to and from cmd.exe windows but the process itself is, well, clunky!
Old School:
Right-Click and chose "Mark:
Select your "selection".
Hit "Enter" to copy.
Proceed to Paste!
That's how I've done it until today :)
New School:
Do this once and you'll find your life with a few clicks each time…
Click on the Menu Button to the top left of the window.
Click on "Properties".
There's the magic option I never noticed or payed attention to...
Now, all you have to do is just click and select your "selection".
Hit "Enter" and paste away!
If you already knew this, please let me have this moment of joy. I'm very excited when I learn something new :)
Cheers!!
So, in the spirit of sharing and in my utter innocence and assumption that most of my readers are as clueless as I am, I'd like to share something new I learned about CMD.EXE :) A certain new friend of mine, whose name I will not mention except that his initials are NC, enlightened me today.
Often enough, we need to copy and paste content to and from cmd.exe windows but the process itself is, well, clunky!
Old School:
Right-Click and chose "Mark:
Select your "selection".
Hit "Enter" to copy.
Proceed to Paste!
That's how I've done it until today :)
New School:
Do this once and you'll find your life with a few clicks each time…
Click on the Menu Button to the top left of the window.
Click on "Properties".
There's the magic option I never noticed or payed attention to...
Now, all you have to do is just click and select your "selection".
Hit "Enter" and paste away!
If you already knew this, please let me have this moment of joy. I'm very excited when I learn something new :)
Cheers!!
Friday, August 2, 2013
An Alternative to Oracle Enterprise Managers GoldenGate Plugin
If you've arrived at this page then, most likely you're a victim of the JAGENT :) Not to be confused with Agent J.
I can't tell you how many times I've been burned by it when using it for OEM monitoring. The symptoms, if you care to read, include:
After banging my head against the wall with MOS, who told me that they're getting a lot reports from customers regarding JAGENTs, I decided to take things into my own hands. Fortunately, I am savvy enough with OEM and its Metric Extension architecture to build my own "plugin".
As a disclaimer to all perl guru's, the attached script is my first attempt at writing in the language. I consider myself quite humle and wide open to improvements and suggestions.
If you're like me, then you want to consolidate your work into concise folder. I chose the GoldenGate installation directory.
Copy monitor_gg.pl and ggsci_syntax.txt into this directory.
The key parameters to consider are at the beginning of the script.
Reset the directory variables to your respective environment (by the way, the script works on Windows as well as long as you point it to perl binaries).
Give it a test to make sure that it works. My output looks is the following:
It is concatenating the strings to represet the current object type, status, lag at checkpoint (seconds), and time since last check point (seconds) values.
Here on, you simply need to create a Metric Extension (for a Host Target Type) within OEM and configure your incident rule sets accordingly. If I had more time, I would show you how I set up mine but I didn't capture any screenshots. The documentation for it pretty straightforward and the GUI itself is intuitive.
On my "All Metrics" for the host, I see the following:
There's plenty to improve on in the script. Since I have an active ticket with Oracle Support, unless they come back with a solution quickly, I will continue to improve on the attached script.
Hope this was helpful for at least some of you :)
I can't tell you how many times I've been burned by it when using it for OEM monitoring. The symptoms, if you care to read, include:
- Hanging Extract and Pumps
- BDB datastore corruptions
The solution, most times, to the above problems was to either rebuild the datastore or remove the JAGENT completely.
After banging my head against the wall with MOS, who told me that they're getting a lot reports from customers regarding JAGENTs, I decided to take things into my own hands. Fortunately, I am savvy enough with OEM and its Metric Extension architecture to build my own "plugin".
As a disclaimer to all perl guru's, the attached script is my first attempt at writing in the language. I consider myself quite humle and wide open to improvements and suggestions.
If you're like me, then you want to consolidate your work into concise folder. I chose the GoldenGate installation directory.
Copy monitor_gg.pl and ggsci_syntax.txt into this directory.
The key parameters to consider are at the beginning of the script.
Reset the directory variables to your respective environment (by the way, the script works on Windows as well as long as you point it to perl binaries).
Give it a test to make sure that it works. My output looks is the following:
It is concatenating the strings to represet the current object type, status, lag at checkpoint (seconds), and time since last check point (seconds) values.
Here on, you simply need to create a Metric Extension (for a Host Target Type) within OEM and configure your incident rule sets accordingly. If I had more time, I would show you how I set up mine but I didn't capture any screenshots. The documentation for it pretty straightforward and the GUI itself is intuitive.
On my "All Metrics" for the host, I see the following:
There's plenty to improve on in the script. Since I have an active ticket with Oracle Support, unless they come back with a solution quickly, I will continue to improve on the attached script.
Hope this was helpful for at least some of you :)
Subscribe to:
Posts
(
Atom
)