Showing posts with label cloudera. Show all posts
Showing posts with label cloudera. Show all posts

Wednesday, October 14, 2015

Data as a Service: JBoss Data Virtualization and Hadoop powering your Big Data solutions

Guest blog by Syed Rasheed, Senior Product Marketing Manager
Twitter @Junooni, eMail srasheed@redhat.com


Red Hat and Cloudera, announce the formation of a strategic alliance. From JBoss perspective, the key objective of the alliance is to leverage big data enterprise-wide and not let Hadoop become another data silo. Cloudera combined with Red Hat JBoss Data Virtualization integrates Hadoop with existing information sources including data warehouses, SQL and NoSQL databases, enterprise and cloud applications, and flat and XML files. The solution creates business-friendly, reusable and virtual data models with unified views by combining and transforming data from multiple sources including Hadoop. This creates integrated data available on-demand for external applications through standard SQL and web services interfaces.
The reality at vast majority of organization is that data is spread across too many applications and systems. Most organizations don’t know what they’ve lost because their data is fragmented across the organization. This problem does not go away just because an organization is using big data technology like Hadoop; in fact, they get more complicated. Some organizations try to solve this problem by hard coding the access to data stores. This simple approach inefficiently breaks down silos and brings lock-in with it. Lock-in makes applications less portable, a key metric for future proofing IT. This approach also impedes organizational agility because hard coding data store access is time consuming and makes IT more complex, incurring technical debt. Successful business need to break down the data silos and make data accessible to all the applications and stakeholders (often a requirement for real time contextual services).
redhat-jboss-datavirt
A much better approach to solving this problem is abstraction through data virtualization. It is a powerful tool, well suited for the loose coupling approach prescribed by the Modern Enterprise Model. Data virtualization helps applications retrieve and manipulate data without needing to know technical details about each data store. When implemented, organizational data can be easily accessed using a simple REST API or via familiar SQL interface.
Data Virtualization (or an abstracted Data as a Service) plugs into the Modern Enterprise Platform as a higher-order layer, offering the following advantages:
  • Better business decisions due to organization wide accessibility of all data
  • Higher organizational agility
  • Loosely coupled services making future proofing easier
  • Lower cost
Data virtualization is therefore a critical part of the big data solution. It facilitates and improves the use of big data in the enterprise by:
  • Abstracting big data into relational-like views
  • Integration with existing enterprise sources
  • Adding real time query capabilities to big data
  • Providing full support for standard based interfaces like REST and OData in addition JDBC and ODBC.
  • Adding security and governance to the big data infrastructure
  • Flattening data siloes through a unified data layer.
Want to learn more, download, and get started with JBoss Data Virtualization visit http://www.jboss.org/products/datavirt
Data Virtualization by Example https://github.com/datavirtualizationbyexample
Interested in community version then visit http://teiid.jboss.org/

Be Sociable, Share!

Tuesday, September 29, 2015

Red Hat and the Strata+Hadoop World Conference

This week I am at the Strata plus Hadoop World Conference in New York City which is presented by O'Reilly and Cloudera.  Red Hat has a booth and Product highlights include JBoss Data Virtualization, Red Hat Storage and Red Hat Enterprise Linux.  Take a look at some of the information being highlighted below:


The theme of the conference is Make Data Work.  The Keynotes will be streamed live during the conference on Wednesday, September 30 and Thursday, October 1.  The live stream schedule is below:

Wednesday, September 30

8:45am - 8:50am
 
Roger Magoulas @rogerm (O'Reilly Media), Doug Cutting @cutting(Cloudera), Alistair Croll @acroll (Solve For Interesting)
8:50am - 9:05am
 
Mike Olson @mikeolson (Cloudera)
9:05am - 9:15am
 
AnnMarie Thomas @amptMN (School of Engineering and Schulze School of Entrepreneurship, University of St. Thomas)
9:15am - 9:25am
 
Joseph Sirosh (Microsoft)
9:25am - 9:30am
 
Ron Kasabian (Intel), Michael Draugelis @mdraugelis (Penn Medicine)
9:30am - 9:35am
 
Tim Howes @howes28 (ClearStory Data)
9:35am - 9:40am
 
Jim McHugh (Cisco)
9:40am - 9:50am
 
Joy Johnson @joyjohnson (AudioCommon)
9:50am - 10:00am
 
David Boyle @beglen (BBC Worldwide)
10:00am - 10:10am
 
DJ Patil @dpatil (White House Office of Science and Technology Policy)
10:10am - 10:25am
 
Katherine Milkman @Katy_Milkman (Wharton School at the University of Pennsylvania)
10:25am - 10:30am
 
Ben Lorica @bigdata (O'Reilly Media)
10:30am - 10:45am
 
Jeff Jonas @jeffjonas (IBM)

Thursday, October 1

8:45am - 8:50am
 
Roger Magoulas @rogerm (O'Reilly Media), Doug Cutting @cutting(Cloudera), Alistair Croll @acroll (Solve For Interesting)
8:50am - 9:00am
 
Doug Wolfe (CIA)
9:00am - 9:10am
 
Daniel Goroff @DGoroff (Alfred P. Sloan Foundation)
9:10am - 9:20am
 
Jack Norris @Norrisjack (MapR Technologies)
9:20am - 9:25am
Ben Werther @bwerther (Platfora)
9:25am - 9:30am
Paul Kent @hornpolish (SAS)
9:30am - 9:45am
 
Farrah Bostic @farrahbostic (The Difference Engine)
9:45am - 9:50am
 
Shivakumar Vaithyanathan (IBM)
9:50am - 10:00am
 
Jake Porway (DataKind)
10:00am - 10:20am
 
Maciej Ceglowski @pinboard (Pinboard.in)
10:20am - 10:40am
 
Maria Konnikova (The New Yorker | Mastermind)
10:40am - 10:45am
 
Roger Magoulas @rogerm (O'Reilly Media), Doug Cutting @cutting (Cloudera), Alistair Croll @acroll (Solve For Interesting)

Tuesday, July 28, 2015

What is the Hadoop Ecosystem?

https://www.facebook.com/hadoopers

In some of our Articles and Demos we have examples of JBoss Data Virtualization (Teiid) using Hadoop as a Data Source through Hive.  When creating examples of Data Virtualization with Hadoop Environments such as Hortonworks Data Platform, Cloudera Quickstart, etc. there are alot of open source projects included.  I wanted to highlight some of those so that you have an overview of the Hadoop Ecosystem.  You can find the information below as well as more projects detail in the ecosystem from the hadoop ecosystem table.

Map Reduce - MapReduce is a programming model for processing large data sets with a parallel, distributed algorithm on a cluster. Apache MapReduce was derived from Google MapReduce: Simplified Data Processing on Large Clusters paper. The current Apache MapReduce version is built over Apache YARN Framework. YARN stands for “Yet-Another-Resource-Negotiator”. It is a new framework that facilitates writing arbitrary distributed processing frameworks and applications. YARN’s execution model is more generic than the earlier MapReduce implementation. YARN can run applications that do not follow the MapReduce model, unlike the original Apache Hadoop MapReduce (also called MR1). Hadoop YARN is an attempt to take Apache Hadoop beyond MapReduce for data-processing.
HDFS - The Hadoop Distributed File System (HDFS) offers a way to store large files across multiple machines. Hadoop and HDFS was derived from Google File System (GFS) paper. Prior to Hadoop 2.0.0, the NameNode was a single point of failure (SPOF) in an HDFS cluster. With Zookeeper the HDFS High Availability feature addresses this problem by providing the option of running two redundant NameNodes in the same cluster in an Active/Passive configuration with a hot standby.
HBase - Google BigTable Inspired. Non-relational distributed database. Ramdom, real-time r/w operations in column-oriented very large tables (BDDB: Big Data Data Base). It’s the backing system for MR jobs outputs. It’s the Hadoop database. It’s for backing Hadoop MapReduce jobs with Apache HBase tables.
Hive - Data Warehouse infrastructure developed by Facebook. Data summarization, query, and analysis. It’s provides SQL-like language (not SQL92 compliant): HiveQL.
Pig - Pig provides an engine for executing data flows in parallel on Hadoop. It includes a language, Pig Latin, for expressing these data flows. Pig Latin includes operators for many of the traditional data operations (join, sort, filter, etc.), as well as the ability for users to develop their own functions for reading, processing, and writing data. Pig runs on Hadoop. It makes use of both the Hadoop Distributed File System, HDFS, and Hadoop’s processing system, MapReduce. Pig uses MapReduce to execute all of its data processing. It compiles the Pig Latin scripts that users write into a series of one or more MapReduce jobs that it then executes. Pig Latin looks different from many of the programming languages you have seen. There are no if statements or for loops in Pig Latin. This is because traditional procedural and object-oriented programming languages describe control flow, and data flow is a side effect of the program. Pig Latin instead focuses on data flow.
Zookeeper - It’s a coordination service that gives you the tools you need to write correct distributed applications. ZooKeeper was developed at Yahoo! Research. Several Hadoop projects are already using ZooKeeper to coordinate the cluster and provide highly-available distributed services. Perhaps most famous of those are Apache HBase, Storm, Kafka. ZooKeeper is an application library with two principal implementations of the APIs—Java and C—and a service component implemented in Java that runs on an ensemble of dedicated servers. Zookeeper is for building distributed systems, simplifies the development process, making it more agile and enabling more robust implementations. Back in 2006, Google published a paper on "Chubby", a distributed lock service which gained wide adoption within their data centers. Zookeeper, not surprisingly, is a close clone of Chubby designed to fulfill many of the same roles for HDFS and other Hadoop infrastructure.
Mahout - Machine learning library and math library, on top of MapReduce.

Also, you can visit the Big Data Insights Page to learn more about Red Hat Products in relation to the Hadoop Ecosystem.