Friday, 24 May 2013

Data Integration Functional Capabilities and Oracle Data Integration Tool Sets

Being in data warehouse and data integration field for many years I always found it difficult to get a complete set of data integration tools as a solution which goes hand in hand. There was always a demand to have service orientated architecture supported a data integration platform to support data movements across heterogeneous sources and targets, a complete data cleansing solution, data profiling, data quality and also offers the real time change data capture and replication to support high availability. Is it too much to expect?   Or It is a market requirement which is driving the excellence in data integration solutions across board.

This blog is an attempt to discuss various project drivers for data integration, what are Oracle Data Integration offerings and its market position on various evaluation factors against the Market Leaders. The information included in this blog is loosely linked with a report from a leading market research company.  

Data Integration Scenarios

The following are the data integration scenarios which data integration platforms to support

Data acquisition for business intelligence (BI) and data warehousing: Extracting data from one or more operational systems, transforming and merging that data, and delivering it to integrated data structures for analytics purposes. BI and data warehousing remain primary source of the demand for data integration tools.

Consolidation & Delivery of master data in support of master data management (MDM): Enabling the consolidation and rationalization of the data representing critical business entities, such as customers, products and employees. Data integration tools can be used to build the data consolidation and synchronization processes.

Data migrations/conversions: Data integration tools are increasingly addressing the data movement and transformation challenges inherent in the replacement of legacy applications and consolidation efforts during mergers and acquisitions.

Synchronization of data between operational applications: Data integration tools provide the ability to ensure database-level consistency across applications, both on an internal and an inter-enterprise basis for software-as-a-service [SaaS] or cloud-resident data sources and in a bidirectional or unidirectional manner.

Interenterprise data sharing: Organizations are increasingly required to provide data to, and receive data from, external trading partners (customers, suppliers, business partners and others). Data integration tools are relevant in addressing these challenges, which often consist of the same types of data access, transformation and movement components found in other common use cases.

Delivery of data services in an SOA context: An architectural technique, rather than a use of data integration itself, data services represent an emerging trend for the role and implementation of data integration capabilities within service-oriented architectures (SOAs). Data integration tools will increasingly enable the delivery of many types of data services.


Functional Capabilities Wish List from Data Integration Platforms

Data Source & Target Support

The data integration platform should to interact with structural, semi-structural and non-structural data types of data structure

·         Relational database, Legacy and non-relational database, file formats ( .txt,.csv ) , XML
·         CRM and SC Management Packaged Applications
·         SaaS and cloud-based applications and sources
·         Industry-standard message formats such as electronic data interchange (EDI), Swift and
·         Health Level Seven International (HL7)
·         HDFS and NoSQL repositories
·         Message queues – Middleware and Standards based products e.g. JMS
·         Emergent data types of a less structured nature, such as email, websites, office productivity tools and content repositories

Mode of Interactions

The data integration platform should support following interactions

·         Bulk acquisition and delivery
·         Granular trickle-feed acquisition and delivery
·         Changed data capture (CDC) — the ability to identify and extract modified data
·         Event-based acquisition (time-based or data-value-based)

Data Delivery

The data integration platform should to provide data to consuming applications, processes and databases in a variety of modes, including:

·         Physical bulk data movement between data repositories
·         Federated views formulated in memory
·         Message-oriented movement via encapsulation
·         Replication of data between homogeneous or heterogeneous database management systems (DBMSs) and schemas

In addition, support for the delivery of data across the range of latency requirements is important, including:

·         Scheduled batch delivery
·         Streaming/near-real-time delivery
·         Event-driven delivery of data based on identification of a relevant event

Data Transformation

The data integration built-in capabilities for achieving data transformation operations of varying complexity, including

·         Basic transformations, such as data type conversions, string manipulations and simple calculations

·         Intermediate-complexity transformations, such as lookup and replace operations, aggregations, summarizations, deterministic matching, and the management of slowly changing dimensions

·         Complex transformations, such as sophisticated parsing operations on free-form text and rich media

·         In addition, the tools must provide facilities for developing custom transformations and extending packaged transformations.
  
Metadata Management and Data Modelling

The data integration should support metadata management and data modelling requirements as below

·         Automated discovery and acquisition of metadata from data sources, applications and other tools
·         Data model creation and maintenance
·         Physical to logical model mapping and rationalization
·         Defining model-to-model relationships via graphical attribute-level mapping
·         Lineage and impact analysis reporting, via graphical and tabular format
·         An open metadata repository, with the ability to share metadata bi-directionally with other tools
·         Automated synchronization of metadata across multiple instances of the tools
·         Ability to extend the metadata repository with customer-defined metadata attributes and relationships
·         Documentation of project/program delivery definitions and design principles in support of requirements definition activities
·         Business analyst/end-user interface to view and work with metadata

Design and Development Environment

The data integration platform should facilitate for enabling the specification and construction of data integration processes such as

·         Graphical representation of repository objects, data models and data flows
·         Workflow management for the development process, addressing requirements such as approvals and promotions
·         Granular, role-based and developer-based security
·         Team-based development capabilities, such as version control and collaboration
·         Functionality to support reuse across developers and projects, and to facilitate the identification of redundancies
·         Support for testing and debugging

Data Governance

The data integration platform should support mechanisms to help the understanding and assurance of data quality over time along with interoperability

·         Data profiling tools
·         Data Mining tools
·         Data quality tools


Deployment Options and Runtime Platform

Breadth of support for the hardware and operating systems on which data integration processes may be deployed, and the choices of delivery model; specifically:

·         Mainframe environments, such as IBM z/OS and z/Linux
·         Midrange environments, such as IBM System i (formerly AS/400) or HP Tandem
·         Unix-based environments
·         Windows environments
·         Linux environments
·         Traditional on-premises (at the customer site) installation and deployment of software
·         Hosted off-premises software deployment (SaaS model)
·         Server virtualization (support for shared, virtualized implementations)
·         Parallel distributed processing (such as Hadoop, MapReduce)


Operations and Administration


The data integration toolset should facilities for enabling adequate ongoing support, management, monitoring and control of the processes implemented via tools such as

·         Error-handling functionality, both predefined and customizable
·         The monitoring and control of runtime processes, both via functionality in the tools and interoperability with other IT operations technologies
·         The collection of runtime statistics to determine use and efficiency, as well as an application style interface for visualization and evaluation
·         Security controls, for both data "in flight" and administrator processes
·         A runtime architecture that ensures performance and scalability

Architecture and Integration

The data integration components should have degree of commonality, consistency and interoperability between them such as follows

·         A minimal number of products (ideally one) supporting all data delivery modes
·         Common metadata (a single repository) and/or the ability to share metadata across all components and data delivery modes
·         A common design environment to support all data delivery modes
·         The ability to switch seamlessly and transparently between delivery modes (bulk/batch vs. granular real-time vs. federation) with minimal rework
·         Interoperability with other integration tools and applications, via certified interfaces and robust APIs
·         Efficient support for all data delivery modes, regardless of runtime architecture type (centralized server engine versus distributed runtime)


Service Enablement (SOA Enabled)

The data integration platform or toolset must exhibit service-oriented characteristics and provide support for SOA deployments

·         The ability to deploy all aspects of runtime functionality as data services
·         Management of publication and testing of data services
·         Interaction with service repositories and registries
·         Service enablement of development and administration environments, so that external tools and applications can dynamically modify and control the runtime behaviour of the tools

Oracle Data Integration Platform

The Oracle Data Integration Platform schematic is shown below.



Does it satisfy the functional capabilities wish list? We will discuss in next blog. I hope you find this shared information useful. 

Monday, 20 May 2013

OBIEE 11g 11.1.1.6.10 Patch Set VS OBIEE 11g 11.1.1.7 Release

Recently Oracle Announced OBIEE 11g 11.1.1.7 for GA, meanwhile the OBIEE 11g 11.1.1.6.10 patch set is also released.

Details of the OBIEE 11g 11.1.1.6.10 Patch Set can be found at following url



You can get the list of bug Fixed in OBIEE 11g 11.1.1.7 Release (OBIEE, BI Publisher and Essbase 11.1.1.2.3)  by following Oracle Support Link.


What should Project do? 

We have got the same problem; we compare these two options on various factors such as stability, number of important bug fixed related to project implementation components, migration efforts  and new features and benefits offered etc.

Finally we decided to go with patch as in terms of bug/issues and stability we find the going with the patch set is a safe option. Looking for the migration opportunity on the project as it is just impossible to ignore the new features offered by 11.1.1.7.

I belive decision should be driven by project requirements and business goals for the implementations.

I hope you find this information useful.


OBIEE 11g: Bugs Fixed in 11.1.1.6.10 Patch Set Oracle Support Web Notification

Applies to:


Business Intelligence Server Enterprise Edition - Version 11.1.1.6.0 to 11.1.1.6.9 [Release 11g]
Business Intelligence Suite Enterprise Edition - Version 11.1.1.6.0 to 11.1.1.6.9 [Release 11g]
Oracle Exalytics Software - Version 1.0.0.0.0 to 1.0.0.2.0 [Release 1.0]
Information in this document applies to any platform.

Details


The Business Intelligence Enterprise Edition 11.1.1.6.10 patch set has been released and is available to download from My Oracle Support.

Per the patch readme: 
  • This patch set is available for all customers who are using Oracle Business Intelligence Enterprise Edition 11.1.1.6.0, 11.1.1.6.1, 11.1.1.6.2, 11.1.1.6.2 BP1, 11.1.1.6.4, 11.1.1.6.5, 11.1.1.6.6, 11.1.1.6.7, 11.1.1.6.8 and 11.1.1.6.9. 
  • Oracle Exalytics customers must only install this Oracle Business Intelligence patch set if it is certified for the specific Oracle Exalytics patch or patch set update that they are applying. For more information see Oracle Fusion Middleware Installation and Administration Guide for Oracle Exalytics In-Memory Machine and the Oracle Exalytics certification information.
The Oracle Business Intelligence 11.1.1.6.10 patch set is comprised of the following patches:
·         Patch 16504136 - 11.1.1.6.10 (1 of 7) Oracle Business Intelligence Installer (BIINST)
·         Patch 16504143 - 11.1.1.6.10 (2 of 7) Oracle Real Time Decisions (RTD)
·         Patch 16504148 - 11.1.1.6.10 (3 of 7) Oracle Business Intelligence Publisher (BIP)
·         Patch 16504154 - 11.1.1.6.10 (4 of 7) Oracle Business Intelligence ADF Components (BIADFCOMPS)
·         Patch 16504156 - 11.1.1.6.10 (5 of 7) Enterprise Performance Management Components Installed from BI Installer 11.1.1.6.x (BIFNDNEPM)
·         Patch 16427939 - 11.1.1.6.10 (6 of 7) Oracle Business Intelligence: (OBIEE)
·         Patch 16504161 - 11.1.1.6.10 (7 of 7) Oracle Business Intelligence Platform Client Installers and MapViewer
Note:
  • The Readme files for the above patches describe the bugs fixed in each patch, and any known bugs with the patch.
  • This patch is cumulative, and therefore, contains all of the fixes included in the earlier 11.1.1.6.2, 11.1.1.6.4, 11.1.1.6.5, 11.1.1.6.6, 11.1.1.6.7, 11.1.1.6.8 and 11.1.1.6.9 patch sets.
  • However, lists of fixes from included patch sets need to be looked up in the respective patches' readme files, and are not included in the above patches' readme files.
  • The instructions to apply the above patches are identical, and are contained in the readme file for patch 16504136.
  • Please bear in mind, that the readme states to apply patch 13952743 for JDeveloper, too.
List of bugs are as follows....


Patch 16504136
13404296 BI PROVISIONING FACADE MUST TRIM RPD IN INSTANCE FOR SELECTED FA OFFERINGS
13485655 T2P: ERROR MSG EXECUTING WLST SCRIPT APPEARS DURING BIINSTANCE PASTECONFIG
13504454 BLK: BIINSTANCE COPYCONFIG FAILURE: ESSBASE MAXL LOGIN IS FAILING 
13510938 NEED OFFICIAL WAY TO INVOKE _CONFIGURERPD
13560245 BIDOMAIN/CONFIG/JDBC NOT FOUND
13565722 BLK: BI T2P PASTECONFIG ERRORED OUT WITH UNABLE TO PARSE JMS_WEBLOGIC_JNDI_URL
13589944 BI T2P: PASTECONFIG INSTANCE IS FAILING WITH JAVA.LANG.UNSATISFIEDLINKERROR
13619973 INCLUDE ESSSERVER HOST NAME IN THE MOVEPLAN
13628293 EMPTY PROCUREMENT DASHBOARD IN RC4
13683204 STRESS:FA:CRM ERROR RESPONSE[NQSERROR: 43082] ILLEGAL OPERATION ATTEMPTED CLOSED
13240778 QA: ACCESS: ERE: RGRN: ERROR ACCESSING LOV VIA KEY; CAN'T SELECT VIA KEY
13347251 QA: UNABLE TO SEE ERE IN "DYNAMIC" PAGE IN SAFARI, CHROME
13360752 UNABLE TO LOAD ALL EXTENSION COLUMNS FOR CUSTOMER IN RTD STUDIO
13408870 QA: SDCONFIG 3X VALUES COPIED TO 11G, BUT NOT USED THERE
13409526 QA: UPDATE DOC, COPYRTDDATA_README.TXT FOR STOP/START 11G
13472560 QA:ACCESS:HIGH CONTRAST SETTING APPEARS CLUTTERED, SOME ITEMS NOT VISIBLE
13506987 COPYRIGHT INFORMATION NEEDS TO BE EXTENDED TO 2012
13537093 BACKPORT BUG 13537093 TO 11.1.1.6.2
13539353 BACKPORT BUG 13539353 TO 11.1.1.6.2
13567294 REMOVED 1ST VALUE IN RULE BECOMES NAN
10022187 ACC: SELECTED ITEM IN FOLDERS PANE GETS LOST AFTER USING UP/DOWN ARROW TOO MUCH
10046249 ACC: SCHEDULE PAGES HAVE "GRIDTABLE" TABLES THAT CAN'T BE NAVIGATED
11678983 LDAP CONFIG ASSUME ADMIN IS DEFINED UNDER DISTINGUISHED NAME FOR USERS
11906714 QA: TIMEZONE NOT IN SYNC IN JOB MANAGER - JOB HIISTORY
12312461 NLS:MB DATA GARBLED IN JOB HISTORY USING SQL SERVER
12542914 ACC: REPORT VIEWER STRUCTURE HAS ERRORS - NO IFRAME AND NO LANG ATTRIBUTE
12645567 NLS:WRONG WEEK OF YEAR FOR ORACLE DATE FORMAT CONTAINING WW
12825409 INTERACTIVE VIEWER CONDITIONAL FORMATTING FAILS FOR NON DISPLAY COLUMN
12877824 WEBSERVICE API - GETSCHEDULEDREPORTSTATUS ALWAYS RETURNS JOB STATUS AS SCHEDULED
12912473 BI PUBLISHER 11G SHARE REPORT XMODE=3 AND XMODE=4 NOT WORKING FOR PDF OUTPUT
13365954 ERROR WHILE OVERWRITING AN ANALYSIS VIA BI COMPOSER
13520338 STRESS:FA:CRM:SPD HEAVY CONTENTION ON FACTORYMANAGER.CREATEMODELOBJECT
13597290 11.1.1.7.0 ESSBASE DATA CONTROL CODE IS PRINTING UNWANTED DEBUG MESSAGES
13733098 REPORTS AND ANALYTICS, CANNOT SAVE NEWLY CREATED REPORT
13789351 RUP2 ST3 (PS5 RC4) - NPE WHILE TESTING OL FLOW IN KOREAN LANGUAGE
13858848 BIADFCOMPS - ANALYSIS CREATED USING COMBINE UNION IS NOT EDITABLE IN BI COMPOSER
13958089 BI COMPOSER: CAN NOT SAVE ANALYSIS AFTER FAILOVER WLS MANAGE SERVER
13973674 MAIN: NAMECLASHEXCEPTION WHEN MULTIPLE INSTANCES OF ESSBASEDC ARE LOADED
14081217 STRESS:FA:CRM:SPD BIADAPTERDCSERVICE HOLDING LARGE IN HEAP CAUSING OOM
14183506 ALWAYS SAME NPE AND STACKTRACE NO MATTER THE PROBLEM IS 
10623654 FA: NLS:PS1:DATE AND TIME NOT LOCALIZED IN FR SCHEDULED BATCHES DETAIL INFO.
11652873 FA: FR ANNOTATION VIEW PANE HAS PERMISSION ISSUES FOR ANNOTATION ON TEXT OB
11735177 REHOSTING TOOL IS NOT TRANSFERRING PROPER USERNAME IN PARTITION DEFINITION
11924085 UNABLE TO RUN AN ESSBASE CALC SCRIPT IN WEB ANALYSIS
12404364 ESSBASE READ ONLY ACCESS INSUFFICIENT PRIVILEGE WITH PROMPT
12530876 PERFORMANCE OF FR REPORT QUERYING ESSBASE W/LARGE NUMBER OF DYNAMIC CALCULATIONS
12601370 FUSION B17: SAVED QUERY DOES NOT REFRESH, BUT NEW QUERY WORKS ON THOSE MEMBERS
12722750 FA: FF 5.0 GIVES BLANK SCREEN FOR HTML BOOK PREVIEW
12723793 NEED DIFFERENT URL FOR SMARTVIEW INSTALLER IN FUSION MODE
12765822 DMS LOGS LOTS OF ERRORS WHEN ESSBASESERVER1 IS DOWN
10137467 ENABLE "ZOOM TO DATA RANGE "
10165057 QA:THE EDGES OF MAP VIEW GOT TRUNCATED AFTER EXPORTED/PRINTED TO PDF/HTML
10167311 ADD TEST CASES FOR MDSXUDML GENERATOR AND PARSER
10199188 CHARTS X AXIS LABEL IS BLURRED AT 45/60 ANGLE
10391144 WEIRD BEHAVIOUR OF FILE OPEN DIALOG ON WINDOWS 7 COMPUTER
10409969 BH9:QA: TIMESTEN ODBC ERR INCOMPATIBLE FACT IF CONDITION STEP FEDERATED MEASUR
10419667 UT:USAGE TRACKING ERROR TEXT OF FAILED QUERY IS NOT LOGGED
10647655 COLUMN DATA FORMAT IS IGNORED ON WRITEBACK
11657725 MALFORMED URL ERROR WHEN PRINT KPI WATCHLIST TO PDF ON DASHBOARD PAGE
11810127 ADMINTOOL TESTS FAIL: EXTERNALIZATION
13794002 BISHIPHOME 11.1.1.6.2 CUMULATIVE PATCH 1
14054848 RUP4: BISHIPHOME CLIENT INSTALL 11.1.1.6.3 PATCH 1
14533251 REL6: BISHIPHOME CLIENT INSTALL 11.1.1.6.31 PATCH 2
14646289 REL6: BISHIPHOME CLIENT INSTALL 11.1.1.6.31 PATCH 3
14672002 REL6: BISHIPHOME CLIENT INSTALL 11.1.1.6.31 PATCH 4
14711062 REL6: BISHIPHOME CLIENT INSTALL 11.1.1.6.31 PATCH 5
14712346 11.1.1.6.5 CLIENT INSTALLER AND MAPVIEWER PATCH
14740701 REL6: BISHIPHOME CLIENT INSTALL 11.1.1.6.31 PATCH 6
15839347 11.1.1.6.6 CLIENT INSTALLER AND MAPVIEWER PATCH
15860153 REL6: BISHIPHOME CLIENT INSTALL 11.1.1.6.31 PATCH 7
I hope you find this copy and pasted information useful.