The Open Universe

Should DevOps be DevOpsSec

2015-11-20T15:44:00.000-05:00

Neil MacDonald, a Gartner Fellow, published a blog on Security in DevOps. I like his comment on a secure system,

A well-designed, developed and managed system is the foundation of a secure system.

He has published a research note on DevOpsSec: Creating the Agile Triangle. You can find his original blog here.

DevOps seeks to bridge the development and operations divide through the establishment of a culture of trust and shared interest among individuals in these previously siloed organizations. However, this vision is incomplete without the incorporation of information security, which represents yet another silo in IT. Breakdowns in communications and processes across development, operations and security are the root cause of the vast majority of critical system downtime, including downtime caused by breaches in security. For example, Gartner research shows that 75% of successful attacks occur against previously known vulnerabilities for which a patch or secure configuration standard was already available (actually, this used to be about 90%, but advanced and targeted attacks have changed the equation).

Conventional wisdom believes the agile nature of the DevOps vision is fundamentally at odds with the historically static and cumbersome nature of information security. I disagree. I believe that security can support a unified vision of DevOpsSec, but to do this, information security must change in multiple ways including security infrastructure becoming more adaptive andprogrammable and making information security representation an integral part of DevOpsSec teams from the genesis of new applications and services.

I’ve just published a research note for clients DevOpsSec: Creating the Agile Triangle that makes the argument for DevOpsSec and outlines the major areas of change for information security to support a unified DevOpsSec vision. My colleague, Cameron Haight, from the IT Operations side of Gartner research joined me on the research note. He has pioneered much of the research on DevOps for Gartner and increasingly he is being asked how DevOps can be adopted without sacrificing security. Increasingly, I am being asked how to rationalize the agile nature of DevOps with the need for security testing. Together, we teamed up to deliver the first in a series of research notes on how to deliver DevOpsSec.

Development, operations and security are fundamentally intertwined. A well-designed, developed and managed system is the foundation of a secure system. DevOps must evolve to a new vision of DevOpsSec that balances the need for speed and agility of enterprise IT capabilities with the enterprise need to protect critical assets, applications and services.

Webinar: How to become a data-driven organization to achieve more and gain a competitive edge

2015-10-30T16:10:00.000-04:00

Data-driven companies share characteristics that help them achieve more and gain a competitive advantage within their industry. The importance of data-driven thinking is not new, but what does it mean in practice?

Red Hat and Mammoth Data deliver a modern, cloud-ready architecture using Hadoop and Spark to transform your organization into a data-driven one.

Join this webinar to learn about:

What a data driven company is and what the challenges are to becoming one
Moving to real time—the stages of data-driven transformation
Data consolidation and analytics
Computer-aided decision making
Real-time decision making:

The semantic web, natural language, and “everything-as-a-service”
Modern data architecture and solutions

A case study, using a real-world situation, will also be a part of this webinar.

Speakers:

Syed Rasheed, sr. product marketing manager, Red Hat
Andrew C. Oliver, president and founder, Mammoth Data

Join the live event:
Time zone converter

Tuesday, November 17, 2015 | 11 a.m. EDT | 8 a.m. PDT

Using the Unified Push Server with BPMS

2015-10-21T03:35:00.000-04:00

During this past Red Hat Summit I worked on a session with Mobile, Fuse and BPMS. It was a great pleasure working with Phil Simpson, Javier Perez and Maggie Hu to build and present the demo. Below I walk through setting up and using the Push Notification Server. For information on submitting claims through the mobile application and the adjudicator review you can review Maggie's Blog on Red Hat Mobile Application Platform - Connecting to JBoss BPMSuite REST.

Red Hat Summit 2015 -
Photo Taken by Wanja Pernath

This article is part of a series we're putting together for the Build an Enterprise Application in 60 Minutes with JBoss Middleware session at the Summit. The original slide decks can be found below from Maggie on Slideshare. Please take a look at the slide decks to get a better idea on the use case we're working with.

Basically, we're allowing people to submit claims quickly from their mobile devices with some basic information when an auto accident happens. The claim will get submitted to the insurance company, an adjudicator will review the claim, set the settlement amount, and complete the claim. Then the policy holder will receive a push notification on their mobile device on the settlement amount.

Red Hat Summit 2015 - Build an Enterprise Application in 60 Minutes with JBoss Middleware

What is Push Notification and its benefits?

A push notification is a message or alert delivered by a centralized server (on premise or cloud) to a device. Unlike pull notifications, in which the client must request information from a server, push notifications originate from a server. Typically, the end user must opt-in to receive alerts; opt-in usually takes place during the install process and end users are provided with a way to manage alerts if they change their minds later on.

An important advantage of push notifications in mobile computing is that the technology doesn't require specific applications on a mobile device to be open in order for a message to be received. This allows a smartphone to receive and display social media or text message alerts even when the device's screen is locked and the social media application that is pushing the notification is closed.

The Red Hat Mobile Unified Push Server (Aerogear) can be seen as a broker that distributes push messages to different 3rd party Push Networks. The graphic below gives a little overview:

One PushApplication and at least one mobile platform variant must be created.
The variant credentials that are generated and stored by the UnifiedPush Server must be added to the mobile application source, enabling the application to register with the UnifiedPush Server once it is installed on mobile devices.
Sending a push message can happen in different ways: The AdminUI can be used to send a (test) message to registered devices. However, in a real-world scenario the Push Notification Message request is triggered from a backend application, which sends its requests using the Sender API. Different SDKs for different languages are supported.
The push request is then translated into platform specific details for the required variant Push Network. The Dashboard of the AdminUI gives a status report if a message is sent to the Push Network.
The UnifiedPush Server does not directly deliver the message to the mobile device. This is done by the appropriate variant Push Network. Note: There can be latency in the actual delivery. Most Push Networks, such as APNs or GCM, do not guarantee to deliver messages to mobile devices.

Setting Up the Unified Push Server on Openshift

The Unified Push Server allows developers to send native push messages to Apple’s Push Notification Service (APNS) and Google’s Cloud Messaging (GCM). It features a built-in administration console that makes it easy for developers to create and manage the push related aspects of their applications for any mobile development environment. Includes client SDKs (iOS, Android, & Cordova), and a REST based sender service with an available Java sender library.

In our example we setup the UPS on Openshift instead of on premise and we do this through the Openshift Application Manager. First we need a Openshift account. You can sign up for a free account for 3 small gears.

Next we will create an openshift application.

We will use the Unified Push Server 1.0 Cartridge which will also add MySQL to the Application. We can take the defaults and click Create Application Button. After creating the application a credentials screen is displayed with URLs and credentials.

Now we can sign onto the console to create applications, variants, etc and monitor messages. Let's open the dashboard for the Push Server, ie https://jbossunifiedpushtest2-ossmentor.rhcloud.com/ag-push, and login with admin/admin which will bring up the dasboard main screen.

Now we want to add an application to receive the push notifications. We click on applications and the create application button. Once created we can look at the variants. This shows the application ID and Master Secret for the application which we will use in our example.

At this point we can create variants for Android and iOS. For this first example we will create an application but won't use variants and register devices. We will just show the notification getting to the push notification server. Now that the application is created let's do a quick test.

Test sending a message to the UPS

I used the Advanced REST Client in Chrome to test the message. I setup the project with the URL, the Authorization and Content Type as shown below. I put in some sample data for the notification.

After clicking on send I get a Job submitted response and can lookup the message in the dashboard.

Now let's move onto how we can make the call from BPMS.

Sending a notification message from BPMS to the UPS

First in our Business Process we add the message in a script task to the Response Body for the REST call.

kcontext.setVariable("requestBody","{\"ttl\":3600,\"message\":{\"alert\":\"Your Payment Amount is:"+kcontext.getVariable("paymentAmount")+"\",\"action-category\":\"some value\", \"sound\":\"default\",\"badge\":2,\"content-available\":true},\"simple-push\":\"version=123\"}");

Next in our Business Process we added a REST Task to make the call to the UPS to send the claim award message. A couple of items to note on the Task. The first is the data assignments. The username is the Application ID and the password is the Master Secret as seen in the screen shots below. Also note the URL for the domain REST API for UPS in addition to the Basic Authorization.

Note the App settings in addition to the Android Variant settings for Google Cloud messaging.

Also in the REST Task we setup the Data Input and Output as indicated below.

One additional item that was required is the work definition as displayed below.

import org.drools.core.process.core.datatype.impl.type.StringDataType;

import org.drools.core.process.core.datatype.impl.type.ObjectDataType;

[

"name" : "Email",

"parameters" : [

"From" : new StringDataType(),

"To" : new StringDataType(),

"Subject" : new StringDataType(),

"Body" : new StringDataType()

"displayName" : "Email",

"icon" : "defaultemailicon.gif"

[

"name" : "Log",

"parameters" : [

"Message" : new StringDataType()

"displayName" : "Log",

"icon" : "defaultlogicon.gif"

[

"name" : "WebService",

"parameters" : [

"Url" : new StringDataType(),

"Namespace" : new StringDataType(),

"Interface" : new StringDataType(),

"Operation" : new StringDataType(),

"Parameter" : new StringDataType(),

"Endpoint" : new StringDataType(),

"Mode" : new StringDataType()

"results" : [

"Result" : new ObjectDataType(),

"displayName" : "WS",

"icon" : "defaultservicenodeicon.png"

[

"name" : "Rest",

"parameters" : [

"Url" : new StringDataType(),

"Method" : new StringDataType(),

"ConnectTimeout" : new StringDataType(),

"ReadTimeout" : new StringDataType(),

"Username" : new StringDataType(),

"Password" : new StringDataType()

"results" : [

"Result" : new ObjectDataType(),

"displayName" : "REST",

"icon" : "defaultservicenodeicon.png"

]

That was all that is required to send a push notification from BPMS to the Unified Push server. You can see the messages in the UPS console.

All variants associated with the Application in UPS will receive the notification. We will cover the Android application in a follow up article to register the device with Google Cloud Messaging (GCM) and receive the notification.

References:

http://searchconsumerization.techtarget.com/definition/push-notification

https://aerogear.org/docs/unifiedpush/ups_userguide/index/

https://developers.openshift.com/en/xpaas-unified-push.html

Red Hat JBoss Enterprise Application Server (EAP) and the Payment Card Industry (PCI) Data Security Standard

2015-10-19T15:33:00.001-04:00

Our guest blogger this week is Albert T. Wong ([email protected])

The Payment Card Industry (PCI) Data Security Standard (DSS) is a proprietary information security standard for organizations that handle branded credit cards from the major card schemes including American Express, Discover Financial Services, JCB, MasterCard Worldwide and Visa International. The PCI Standard is mandated by the card brands and administered by the Payment Card Industry Security Standards Council.

The Payment Card Industry Data Security Standard (PCI DSS) was developed to encourage and enhance cardholder data security and facilitate the broad adoption of consistent data security measures globally. PCI DSS provides a baseline of technical and operational requirements designed to protect account data. PCI DSS applies to all entities involved in payment card processing—including merchants, processors, acquirers, issuers, and service providers. PCI DSS also applies to all other entities that store, process or transmit cardholder data (CHD) and/or sensitive authentication data (SAD).

The PCI DSS Version 3.1 standard (released in 2015) lists twelve (12) requirements which retailers, online merchants, credit data processors, and other payment related businesses must implement to help protect cardholders and their data. The requirements include technology controls (such as data encryption, virus protection, end-user access control and activity monitoring) as well as required procedures.

Most of the requirements focus on site security and encryption, but some of them apply to securing your applications. The JBoss Enterprise Application Server (EAP) team has produced this technical overview document to assist you in understanding the PCI requirements, determining which requirements apply to JBoss Enterprise Application Server (EAP), and how JBoss Enterprise Application Server (EAP) implements the applicable requirements.

The use of JBoss Enterprise Application Server (EAP) in your electronic commerce site, even if installed and configured correctly, does not guarantee that your site will be PCI compliant. The purpose of this document is to describe the relationship between JBoss Enterprise Application Server (EAP) and the PCI Data Security Standard requirements, not about an entire operating environment. PCI compliance can also impose requirements on other components of your site involved in the storage, processing, or transmission of cardholder data, including firewalls, routers, Web servers, Operating Systems, databases and the web application. PCI compliance remains solely the responsibility of the merchant.

For your reference, here is the outline of the standard:

Build and Maintain a Secure Network

Requirement 1: Install and maintain a firewall configuration to protect cardholder data.

Requirement 2: Do not use vendor-supplied defaults for system passwords and other security parameters.

Protect Cardholder Data

Requirement 3: Protect stored cardholder data.

Requirement 4: Encrypt transmission of cardholder data across open, public networks.

Maintain a Vulnerability Management Program

Requirement 5: Protect all systems against malware and regularly update anti-virus software or programs.

Requirement 6: Develop and maintain secure systems and applications.

Implement Strong Access Control Measures

Requirement 7: Restrict access to cardholder data by business need to know.

Requirement 8: Identify and authenticate access to system components.

Requirement 9: Restrict physical access to cardholder data.

Regularly Monitor and Test Networks

Requirement 10: Track and monitor all access to network resources and cardholder data.

Requirement 11: Regularly test security systems and processes.

Maintain an Information Security Policy

Requirement 12: Maintain a policy that addresses information security for all personnel.

Where to find information about the Payment Card Industry (PCI) Data Security Standard

Payment Card Industry Data Security Standard: https://www.pcisecuritystandards.org/index.shtml

JBoss Enterprise Application Server (EAP) and PCI compliance

The PCI Data Security Standard (DSS) addresses far more than the security of your JBoss Enterprise Application Server (EAP) application. It covers broad security requirements such as virus protection, and restricting physical access to cardholder data.

It is important to recognize the scope of the requirements, and which of them are related to JBoss Enterprise Application Server (EAP).

Control Objective	Relationship
1: Install and maintain a firewall configuration to protect cardholder data.	Related only to PCI DSS
2: Do not use vendor-supplied defaults for system passwords and other security parameters.	Focus area
3: Protect stored cardholder data.	Related only to PCI DSS
4: Encrypt transmission of cardholder data across open, public networks.	Focus area
5: Protect all systems against malware and regularly update anti-virus software or programs.	Related only to PCI DSS
6: Develop and maintain secure systems and applications.	Related only to PCI DSS
7: Restrict access to cardholder data by business need to know.	Focus area
8: Identify and authenticate access to system components.	Focus area
9: Restrict physical access to cardholder data.	Related only to PCI DSS
10: Track and monitor all access to network resources and cardholder data.	Focus area
11: Regularly test security systems and processes.	Related only to PCI DSS
12: Maintain a policy that addresses information security for all personnel.	Related only to PCI DSS

PCI Security Standards Council Notices: Legal Terms and Conditions

Acceptance of a given payment application by the PCI Security Standards Council, LLC (PCI SSC) only applies to the specific version of that payment application that was reviewed by a PA-QSA and subsequently accepted by PCI SSC (the "Accepted Version"). If any aspect of a payment application or version thereof is different from that which was reviewed by the PA-QSA and accepted by PCI SSC – even if the different payment application or version (the "Alternate Version") conforms to the basic product description of the Accepted Version – then the Alternate Version should not be considered accepted by PCI SSC, nor promoted as accepted by PCI SSC.

No vendor or other third party may refer to a payment application as "PCI Approved" or "PCI SSC Approved", and no vendor or other third party may otherwise state or imply that PCI SSC has, in whole or part, accepted or approved any aspect of a vendor or its services or payment applications, except to the extent and subject to the terms and restrictions expressly set forth in a written agreement with PCI SSC, or in a PA-DSS letter of acceptance provided by PCI SSC. All other references to PCI SSC's approval or acceptance of a payment application or version thereof are strictly and actively prohibited by PCI SSC.

When granted, PCI SSC acceptance is provided to ensure certain security and operational characteristics important to the achievement of PCI SSC's goals, but such acceptance does not under any circumstances include or imply any endorsement or warranty regarding the payment application vendor or the functionality, quality, or performance of the payment application or any other product or service. PCI SSC does not warrant any products or services provided by third parties. PCI SSC acceptance does not, under any circumstances, include or imply any product warranties from PCI SSC, including, without limitation, any implied warranties of merchantability, fitness for purpose or non-infringement, all of which are expressly disclaimed by PCI SSC. All rights and remedies regarding products and services that have received acceptance from PCI SSC, shall be provided by the party providing such products or services, and not by PCI SSC or any payment brands.

Addressing the PCI Data Security Standard within JBoss Enterprise Application Server (EAP)

The following topics deal with each of the detailed requirements that pertain to JBoss Enterprise Application Server (EAP). Some of the requirements are directly related to the JBoss Enterprise Application Server (EAP) software package. Other requirements are unrelated, or indirectly relate to the JBoss Enterprise Application Server (EAP) software package. For example, indirect requirements can affect your use of the operating system security features to secure JBoss Enterprise Application Server (EAP) files.

PCI Assessment Services for JBoss Enterprise Application Server (EAP)

There is much more to navigating the PCI standard and the certification procedure than simply installing JBoss Enterprise Application Server (EAP) and making the adjustments we have outlined in the preceding sections. There are significant portions of the standard that, although it applies to your site, does not apply to the software application. To assist you in completely addressing these parts of the standard, Red Hat consulting can assist your site in becoming PCI compliant.

Addressing the PCI Data Security Standard within JBoss Enterprise Application Server (EAP)

The following topics deal with each of the detailed requirements that pertain to JBoss Enterprise Application Server (EAP). Some of the requirements are directly related to the JBoss Enterprise Application Server (EAP) software package. Other requirements are unrelated, or indirectly relate to the JBoss Enterprise Application Server (EAP) software package. For example, indirect requirements can affect your use of the operating system security features to secure JBoss Enterprise Application Server (EAP) files.

For several of the requirements that are related only to PCI compliance (and not to JBoss Enterprise Application Server (EAP)) you are referred directly to the PCI DSS for details. Ensure that you keep up with the rapid pace of changing security requirements.

Requirement 1: Install and maintain a firewall configuration to protect cardholder data

Many parts of requirement 1 such as your wireless network or router setup do not directly relate to JBoss Enterprise Application Server (EAP), but the requirements that relate to your site topology are extremely important. You must construct your JBoss Enterprise Application Server (EAP) site so that you never store cardholder data on internet-accessible systems. Additionally, JBoss Enterprise Application Server (EAP) sites should always use firewalls to separate themselves from the internet, internal networks, and any other system that is accessible to the internet. Ensure that you implement JBoss Enterprise Application Server (EAP) in a 3–tier configuration using the JBoss EAP Reference Architecture (http://www.redhat.com/en/resources/jboss-eap-6-clustering)

Requirement 2: Do not use vendor-supplied defaults for system passwords and other security parameters

Read the JBoss EAP Security Guide for details on changing the system password and system hardening.

Requirement 3: Protect stored cardholder data

Beyond the scope of JBoss EAP.

Requirement 4: Encrypt transmission of cardholder data across open, public networks

Disable SSLv2 and older security encryption on your web server.

Enable Federal Information Processing Standards publication 140-2 (FIPS 140-2) security standard.

Enable National Institute of Standards and Technology (NIST) Special Publications 800-131A (SP 800-131A) security standard.

Requirement 5: Protect all systems against malware and regularly update anti-virus software or programs

Although antivirus software is outside the scope of JBoss Enterprise Application Server (EAP), protecting your servers and network from malicious software should always be a priority for a responsible network administrator.

Requirement 6: Develop and maintain secure systems and applications

Ensure that your store error pages do not display stack traces, either visibly, or in the page source.

As your business needs change, you or your business partners might customize your JBoss Enterprise Application Server (EAP) site. As you do so, you must ensure that the customizations do not compromise your site security. Ensure that your developers understand the requirement to develop secure systems by referring to the PA-DSS and PCI-DSS.

Please also monitor the top 10 list of security threats by the Open Web Appliction Security Project (OWASP)

Requirement 7: Restrict access to cardholder data by business need to know

Read the JBoss EAP Security Guide for details on access control lists.

Requirement 8: Identify and authenticate access to system components
Read the JBoss EAP Security Guide for details on default account policies.

Requirement 9: Restrict physical access to cardholder data

Beyond the scope of JBoss EAP.

Requirement 10: Track and monitor all access to network resources and cardholder data

Ensure that the correct level of logging is enabled. Please see JBoss EAP documentation for more details.

Requirement 11: Regularly test security systems and processes

Beyond the scope of JBoss EAP.

Requirement 12: Maintain a policy that addresses information security for all personnel

Beyond the scope of JBoss EAP.

Webinar: Build your next business application in 60 minutes

2015-10-15T16:12:00.001-04:00

IT is at the center of change in application development as new approaches like DevOps emphasize rapid delivery of features and greater participation by end users and “citizen developers.” Middleware technologies such as integration, business rules, business process management (BPM), mobile application platforms, and Platform-as-a-Service (PaaS) are emerging as important tools for DevOps teams, greatly reducing the traditional coding requirements for new business applications.

Join us for this webinar to learn how two middleware technologies—business process management and mobile app development platforms—can work together in the cloud to support development of process-enabled mobile business apps. Through a real-word use case, we will demonstrate:
Development and deployment of a mobile business app using Red Hat® Mobile Application Platform.
Platform inclusion of captured mobile information into business processes with Red Hat JBoss® BPM Suite.
Integration of automated processes with systems of record with Red Hat JBoss Fuse.
Speakers:
Phil Simpson, senior principal product marketing manager, Red Hat
Maggie Hu, middleware specialist solution architect, Red Hat
Kenny Peeples, JBoss technology evangelist, Red Hat

Join the live event:
Time zone converter
Thursday, October 22, 2015 | 11 a.m. EDT | 8 a.m. PDT

Data as a Service: JBoss Data Virtualization and Hadoop powering your Big Data solutions

2015-10-14T13:45:00.000-04:00

Guest blog by Syed Rasheed, Senior Product Marketing Manager
Twitter @Junooni, eMail [email protected]

Red Hat and Cloudera, announce the formation of a strategic alliance. From JBoss perspective, the key objective of the alliance is to leverage big data enterprise-wide and not let Hadoop become another data silo. Cloudera combined with Red Hat JBoss Data Virtualization integrates Hadoop with existing information sources including data warehouses, SQL and NoSQL databases, enterprise and cloud applications, and flat and XML files. The solution creates business-friendly, reusable and virtual data models with unified views by combining and transforming data from multiple sources including Hadoop. This creates integrated data available on-demand for external applications through standard SQL and web services interfaces.

The reality at vast majority of organization is that data is spread across too many applications and systems. Most organizations don’t know what they’ve lost because their data is fragmented across the organization. This problem does not go away just because an organization is using big data technology like Hadoop; in fact, they get more complicated. Some organizations try to solve this problem by hard coding the access to data stores. This simple approach inefficiently breaks down silos and brings lock-in with it. Lock-in makes applications less portable, a key metric for future proofing IT. This approach also impedes organizational agility because hard coding data store access is time consuming and makes IT more complex, incurring technical debt. Successful business need to break down the data silos and make data accessible to all the applications and stakeholders (often a requirement for real time contextual services).

A much better approach to solving this problem is abstraction through data virtualization. It is a powerful tool, well suited for the loose coupling approach prescribed by the Modern Enterprise Model. Data virtualization helps applications retrieve and manipulate data without needing to know technical details about each data store. When implemented, organizational data can be easily accessed using a simple REST API or via familiar SQL interface.

Data Virtualization (or an abstracted Data as a Service) plugs into the Modern Enterprise Platform as a higher-order layer, offering the following advantages:

Better business decisions due to organization wide accessibility of all data
Higher organizational agility
Loosely coupled services making future proofing easier
Lower cost

Data virtualization is therefore a critical part of the big data solution. It facilitates and improves the use of big data in the enterprise by:

Abstracting big data into relational-like views
Integration with existing enterprise sources
Adding real time query capabilities to big data
Providing full support for standard based interfaces like REST and OData in addition JDBC and ODBC.
Adding security and governance to the big data infrastructure
Flattening data siloes through a unified data layer.

Want to learn more, download, and get started with JBoss Data Virtualization visit http://www.jboss.org/products/datavirt

Data Virtualization by Example https://github.com/datavirtualizationbyexample

Interested in community version then visit http://teiid.jboss.org/

Be Sociable, Share!

Red Hat and the Strata+Hadoop World Conference

2015-09-29T10:57:00.000-04:00

This week I am at the Strata plus Hadoop World Conference in New York City which is presented by O'Reilly and Cloudera. Red Hat has a booth and Product highlights include JBoss Data Virtualization, Red Hat Storage and Red Hat Enterprise Linux. Take a look at some of the information being highlighted below:

The theme of the conference is Make Data Work. The Keynotes will be streamed live during the conference on Wednesday, September 30 and Thursday, October 1. The live stream schedule is below:

Wednesday, September 30

8:45am - 8:50am

Wednesday keynote welcome

Roger Magoulas @rogerm (O'Reilly Media), Doug Cutting @cutting(Cloudera), Alistair Croll @acroll (Solve For Interesting)

8:50am - 9:05am

The next generation

Mike Olson @mikeolson (Cloudera)

9:05am - 9:15am

Playing with, and for, data

AnnMarie Thomas @amptMN (School of Engineering and Schulze School of Entrepreneurship, University of St. Thomas)

9:15am - 9:25am

What 0-50 million users in 7 days can teach us about big data

Joseph Sirosh (Microsoft)

9:25am - 9:30am

Improving Medical Decision Making with Predictive Analytics on Big Data

Ron Kasabian (Intel), Michael Draugelis @mdraugelis (Penn Medicine)

9:30am - 9:35am

The race to modernize BI: What it is and why so urgent?

Tim Howes @howes28 (ClearStory Data)

9:35am - 9:40am

Unleashing the power of big data today

Jim McHugh (Cisco)

9:40am - 9:50am

A Transition to Interactive Music Consumption + Data

Joy Johnson @joyjohnson (AudioCommon)

9:50am - 10:00am

Data vs creativity: The last battleground?

David Boyle @beglen (BBC Worldwide)

10:00am - 10:10am

On reflection: What the White House needs from you

DJ Patil @dpatil (White House Office of Science and Technology Policy)

10:10am - 10:25am

Improving decisions

Katherine Milkman @Katy_Milkman (Wharton School at the University of Pennsylvania)

10:25am - 10:30am

O'Reilly Announcements

Ben Lorica @bigdata (O'Reilly Media)

10:30am - 10:45am

Context Computing

Jeff Jonas @jeffjonas (IBM)

Thursday, October 1

8:45am - 8:50am

Thursday keynote welcome

Roger Magoulas @rogerm (O'Reilly Media), Doug Cutting @cutting(Cloudera), Alistair Croll @acroll (Solve For Interesting)

8:50am - 9:00am

Data science for mission

Doug Wolfe (CIA)

9:00am - 9:10am

Privacy protection and reproducible research

Daniel Goroff @DGoroff (Alfred P. Sloan Foundation)

9:10am - 9:20am

The big data dividend

Jack Norris @Norrisjack (MapR Technologies)

9:20am - 9:25am

The rise of the citizen data scientist

Ben Werther @bwerther (Platfora)

9:25am - 9:30am

Patterns from the future

Paul Kent @hornpolish (SAS)

9:30am - 9:45am

Doing it Wrong: 10 Problems with Qualitative Data

Farrah Bostic @farrahbostic (The Difference Engine)

9:45am - 9:50am

IBM sponsored keynote

Shivakumar Vaithyanathan (IBM)

9:50am - 10:00am

What does it take to apply data science for social good?

Jake Porway (DataKind)

10:00am - 10:20am

Haunted by data

Maciej Ceglowski @pinboard (Pinboard.in)

10:20am - 10:40am

In praise of boredom

Maria Konnikova (The New Yorker | Mastermind)

10:40am - 10:45am

Closing remarks

Roger Magoulas @rogerm (O'Reilly Media), Doug Cutting @cutting (Cloudera), Alistair Croll @acroll (Solve For Interesting)

More detail on Data Virtualization 6.2

2015-09-29T10:30:00.000-04:00

I wanted to send a follow up posting on more detail on JBoss Data Virtualization 6.2 GA.

JDV 6.2 Features

JDV 6.2 is a short release primarily focused on supporting EAP 6.4. Even so, there are some technology updates and other noteworthy improvements:

Support for EAP 6.4
JDV 6.2 is supported with EAP 6.4.3. Note that starting with JDV 6.2, we no longer bundle EAP with the JDV product. The user needs to get EAP from a separate download. This allows JDV customers the flexibility to use existing EAP instances (EAP v 6.4.3 and above) that they already have installed, as well as the ability to move to newer versions of EAP more easily.

JDG 6.4
JDV 6.2 extends support for JDG 6.4 by adding capabilities for the JDG 6.4 Domain-specific Language (DSL) translator utilizing Google Protocol Buffer (protobuf) serialization. This enables you to run SQL queries on data stored in JDG remote caches. With JDV 6.1 only key-based access for remote caches was supported.

Kerberos Delegation Improvements
JDV 6.1 JDBC driver was capable of negotiating a new Kerberos token, but was not capable of propagating an existing Kerberos token. Starting with JDV 6.2, the JDBC driver is capable of participating in Kerberos delegation, if the higher order application stack already negotiated a Kerberos token. This leads the way to true SSO from the client application stack to all the way to the data source, if data source also supports Kerberos delegation. The features to enable Kerberos delegation for Oracle and MS-SQL Server, Hive and Cloudera Impala data sources in addition to JDBC were also added.

Pushdown Query Improvements
Pushdown query capabilities are extended to include salesforce.com, LDAP, JPA and OLAP data sources. In prior versions of JDV, there was the ability to optionally push an entire query to a JDBC data source unchanged. This allows customers the flexibility to be able to use SQL that is proprietary to a data source directly to the source when necessary, without needing to translate it to SQL 92 in the JDV Engine. New for JDV 6.2, we extended this capability beyond JDBC to additional data sources.

Modeshape
In JDV 6.2, we take advantage of EAP security caching in Modeshape to improve performance when an external security domain, like LDAP, is used.

Java 8 MS Access Translator
In JDV 6.1, all the ODBC sources utilized ODBC-JDBC bridge provided in Java. With the removal of ODBC-JDBC Bridge in Java 8, a new translator is introduced to read from the MS Access database.

Apache HBase Resource Adapter (Technical Preview)
Apache HBase is an open source, non-relational, distributed database modeled after Google's BigTable and written in Java. It is developed as part of Apache Software Foundation's Apache Hadoop project and runs on top of HDFS (Hadoop Distributed Filesystem), providing BigTable-like capabilities for Hadoop. In JDV 6.2, we released a resource adapter for Apache HBase that will be officially certified following the JDV 6.2 release.

Tooling Improvements
JDV 6.2 is released with JBDSIS 8.0.3 and JBDS 8.1 (Eclipse Luna)

For JDV 6.2, we improved the usability of Teiid Designer in several areas including:

Resizing of dialogue boxes for smaller screens (particularly useful for demo environments)
Renaming and moving models
REST and SOAP import functionality
Guides View/Cheat Sheet actions
Teiid Connection Importer
General improvements for Mac OS

JDV 6.2 GA is available for download from
- JBoss.org at http://www.jboss.org/products/datavirt/overview/
- Customer Portal at https://access.redhat.com/products/red-hat-jboss-data-virtualization

JDV 6.2 Documentation is available at

- https://access.redhat.com/documentation/en-US/Red_Hat_JBoss_Data_Virtualization/

Technical Blog available at

/2015/09/data-virtualization-62-released.html

Data Virtualization 6.2 Released!

2015-09-18T07:14:00.000-04:00

We are pleased to announce the release of Data Virtualization 6.2. This is a minor release and the highlights are:

EAP 6.4 Support
Teiid Designer usability improvements
Kerberos passthrough for Cloudera, Oracle and MS SQL Server

You can download the product from http://www.jboss.org/products/datavirt/download/ or from the Customer Support Portal (CSP) at https://access.redhat.com/. You will also need to download EAP 6.4 and the EAP 6.4.3 roll up patch to install DV 6.2. Also you can access the updated documentation at https://access.redhat.com/documentation/en-US/Red_Hat_JBoss_Data_Virtualization/6.2/index.html

To install Data Virtualization 6.2:

1. Install EAP 6.4 after downloading from http://www.jboss.org/products/eap/download/
2. Install the EAP 6.4.3 roll up patch from the Customer Support Portal https://access.redhat.com/jbossnetwork/restricted/softwareDownload.html?softwareId=39353
3. Install DV 6.2 after downloading from the above link.

The Data Virtualization 6.2 tooling (Teiid Designer 9.0.3.Final) is available. Note that testers have verified this Designer version with JBDS 8.1.0 (Eclipse Luna).

Download JBDS 8.1.0 : https://www.jboss.org/products/devstudio/download/
Add DV tooling via the JBoss Central panel select the Software/Updates tab, check JBoss Data Virtualization option and click the Install/Update button.

You could also install Teiid Designer 9.0.3 into an 8.0/8.1 version using this update site https://devstudio.redhat.com/updates/8.0/

To help you get started we created a getting started example with CSV and XML sources in a customer context scenario. Run one of 3 options to get you started easily.

Run with JBDS
Run without JBDS
Run with Docker

The Teiid Project is included in the repository so you can take a look at the views and preview data before deployment. Take a look at https://github.com/jbossdemocentral/dv-gettingstarted and get started with Data Virtualization 6.2.

The following table provides a list of data sources and translators that are supported by Red Hat.

Data Source	Translator	Supported DV Version	Driver
Apache Hive 12	-	6.0+	-
Apache Solr	solr	6.1+	-
Cloudera Hadoop	-	6.1+	-
EDS 5.x	teiid	6.0+	-
Files – delimited, fixed length	file	6.0+	-
Generic Datasource-JDBC ansi	jdbc-ansi	6.0+	-
Generic Datasource-JDBC simple (postgresql84)	jdbc-simple	6.0+	-
Google Spreadsheet	-	6.0+	-
Greenplum 4.x	postgresql	6.0+	-
Hortonworks Hadoop	-	6.1.+	-
HSQL (for test/examples only)	-	-	-
IBM DB2 10	db2	6.1+	Universal Driver v4.x
IBM DB2 9.7	db2	6.0+	Universal Driver v4.x
Ingres 10	ingres	6.0+	-
Intel Hadoop	-	6.1+	-
JBoss Data Grid 6.4 (remote client - hotrod)	infinispan-cache-dsl	6.2+	-
JBoss Data Grid 6.4 (library mode)	infinispan-cache	6.0 - post GA, 6.1+	-
LDAP/ActiveDirectory v3	ldap	6.0+	-
Mainframe (CICS,IMS,VSAM)	-	6.0+	-
MariaDB	mysql5	6.1+	-
ModeShape/JCR 3.1	-	6.0+	-
MongoDB 2.2	mongodb 6.0	-	post GA, 6.1+ -
MS Access 2010	-	6.0+	-
MS Access 2013	-	6.0+	-
MS Excel 2010	excel	6.0+	-
MS Excel 2013	excel	6.0+	-
MS SQL Server 2008	sqlserver 6.0+	Microsoft SQL Server	JDBC Driver 4
MS SQL Server 2012	sqlserver 6.0+	Microsoft SQL Server	JDBC Driver 4
MySQL 5.1	mysql5	6.0+	V5.1
MySQL 5.5	mysql5	6.0+	V5.5
Netezza 6.0.2	netezza	6.0+	-
Oracle 10g R2	oracle	6.0+	Oracle JDBC Driver v10
Oracle 11g RAC	oracle	6.0+	Oracle JDBC Driver v11
Oracle 12c	oracle	6.0 - post GA, 6.1+	Oracle JDBC Driver v12
PostgreSQL 8.4	postgresql	6.0+	-
PostgreSQL 9.2	postgresql	6.0+	-
REST/JSON over HTTP	ws	-	-
RHEL 5.5/6 PostgreSQL config	-	6.0+	-
Salesforce.com API 22	salesforce	6.0+	-
SAP Netweaver Gateway (OData)	sap-nw-gateway	6.1+	-
Support SAP Service Registry as a Data Source	-	6.2+	-
Sybase ASE 15	sybase	6.0+	jConnect JDBC3.0 v7
Teradata Express 12	teradata	6.0+	-
Webservices	ws	6.0+	-
XML Files	FILE	6.0+	-

Bossie Awards 2015: The best open source applications

2015-09-17T16:13:00.003-04:00

InfoWorld's top picks in open source business applications, enterprise integration, and middleware. Published from the Bossie Awards by Andrew Oliver.

Teiid is a data virtualization system that allows applications to use data from multiple, heterogeneous data stores. Currently a JBoss project, Teiid is backed by years of development from MetaMatrix and a long history of addressing the data access needs of the largest enterprise environments. I even see uses for Teiid in Hadoop and big data environments.

In essence, Teiid allows you to connect all of your data sources into a “virtual” mega data source. You can define caching semantics, transforms, and other “configuration not code” transforms to load from multiple data sources using plain old SQL, XQuery, or procedural queries.

Teiid is primarily accessible through JBDC and has built-in support for Web services. Red Hat sells Teiid as JBoss Data Virtualization.

-- Andrew C. Oliver

Learn how to build Data Services through this Real World Example

2015-09-09T01:06:00.004-04:00

Learn the full cycle of building data services from installing on premise or in the cloud to consuming the data service. We take you through a real life scenario with a Product scenario in the Cloud with the WebUI, which can also be created on premise. We also feature Red Hat Mobile (Feedhenry) as a Consumer.

Learn Node through our introduction lab for Node.js and Mobility

2015-09-08T04:13:00.000-04:00

Node.js, or Node, applications are event-driven, use a non-blocking I/O API and run asynchronously. Node is being used more and more with Mobile Application Development because of a fast growing Node ecosystem and the ability to use the same platform to develop server and mobile apps. With our Node Lab you will learn the basics of Node and npm as well as essential skills for working with Node. We will also show Node integration with Middleware products such as Data Virtualization and Fuse through a Node.js Cloud App in Red Hat Mobile. In this lab, we will cover topics such as:

Node.js history and overview
Some of the basics of node such as asynchronous i/o and http
How to use and create npm modules -The details of Scope, Scope Chains, Closures, and Garbage Collection
How to create a Node.js Cloud App that makes a rest call with the request module

Red Hat Mobile Application Platform Boot Camp at RHTE

2015-09-05T17:22:00.000-04:00

This week at the APAC Red Hat Tech Exchange we are doing a Red Hat Mobile Application Platform Boot Camp. It is a hands-on, lab-based 2-day course.

Overview: Mobile apps are changing the way enterprises conduct business. Organizations are demanding solutions that enable them to extend existing applications to mobile devices andto better engage with customers, employees and partners. The Red Hat Mobile Application Platform is the latest middleware product from Red Hat. By leveraging Red Hat's strengths in enterprise middleware and Platform-as-a-Service (PaaS) technologies, both Red Hat consultants and partners can enable customers on meeting their growing mobile requirements.

Target Audience: Mobile application developers, middleware solution architects and middleware delivery consultants who wish to learn about the Red Hat Mobile Application Platform and how to develop applications with it. This training is a technical and is designed for an audience with strong development skills.

Red Hat Mobile Application Platform is an intensive hands-on, lab-based 2-day course. In this course, attendees will learn how to:

Create applications using Red Hat Mobile Application Platform
Develop HTML 5 applications using JavaScript and Node.js
Install command-line tools for local development
Create mobile application projects from scratch and import/export existing projects
Integrate MBaaS APIs as reusable services and cloud plugins
Leverage the Data Sync functionality
Develop client-side apps using Apache Cordova hybrid apps and client APIs
Quickly create Form-based apps with a Form Builder

Attendee prerequisites:

Basic knowledge of HTML/CSS (good to have)
Basic knowledge of JavaScript and JavaScript frameworks – ie: Angular.js, Backbone.js (good to have)
Basic knowledge of Node.js (good to have)

To access the lab environment, every attendee will require a PC, complete with Internet Connection and minimum 8GB RAM (mandatory).

Easy Getting Started Example with Data Virtualization for Developers

2015-08-26T14:33:00.000-04:00

Data Virtualization (Teiid) provides Architects, Database Administrators and Developers the ability to build Data Services with federated and aggregated views of data with ease. DV gives them a fast, easy and agile experience in Data Integration. Instead of building connections to individual data sources and transformations for a aggregated view, Developers can use DV for creating Data Abstraction Layers for Data Integration and then utilize many different interfaces for consuming the data.

We have created a simple way for Developers to quickly view the product and view/run a simple project. This gives them a Hello World experience. The Getting Started example revolves around a Customer Context with data from a XML file and a CSV file. The repository has all the instructions to install DV and JBDS, import the project, view data from the views, deploy the Virtual Database and browse to the data. Scripts are also included to automatically install and run DV then test OData urls. This provides the ability to use 1 of 3 options - Option 1 with JBDS and Option 2 without JBDS.

Credentials setup during install
Management Credentials: admin/redhat1!
Datavirtualization Credentials: teiidUser/redhat1!
Datavirtualization Dashboard Credentials: dashboardAdmin/redhat1!

Steps to Run the Demo
OPTION 1 - RUNNING AND PREVIEWING WITHIN JBDS:
STEP 1: Clone the Repository and Download Data Virtualization
-git clone https://github.com/jbossdemocentral/dv-gettingstarted.git
-Download from jboss.org http://www.jboss.org/products/datavirt/download/
-Install Data Virtualization enabling OData

STEP 2: Import, Preview Data and Deploy
-Download JBDS http://www.jboss.org/products/devstudio/download/
-Install JBDS and the Integration Stack https://devstudio.jboss.com/updates/8.0/integration-stack/
-Add and start a new server with the home directory of the installed DV from above
-Verify management and jdbc connections on the Teiid instance
-Refresh the server
-Import the project into JBDS (should have JBDIS installed)
-Verify default teiid instance from actions in Teiid perspective
-Right click on the 3 views/tables and preview the data
-Click on the VDB and execute the selected text SELECT * FROM CustomerContextView.CustomerContextTable;
-Deploy the VDB

STEP 3: Browse the Data Virtualization and the Data throughthe Chrome Browser
-All Data
http://localhost:8080/odata/CustomerContextVDB/CustomerContextView.CustomerContextTable?$format=json
-Specific Entity
http://localhost:8080/odata/CustomerContextVDB/CustomerContextTable('123')?$format=json
-Metadata
http://localhost:8080/odata/CustomerContextVDB/$metadata
-Management Console to view Virtual Database
http://localhost:8080
-Dashboard
http://localhost:8080/dashboard/

As Easy as 1,2,3....

OPTION 2 - STEPS TO RUN WITHOUT JBDS:
STEP 1: Clone the Repository and Download Data Virtualization
-git clone https://github.com/jbossdemocentral/dv-gettingstarted.git
-Download from jboss.org http://www.jboss.org/products/datavirt/download/
Put the Data Virutalization Download, jboss-dv-installer-6.1.0.redhat-3.jar, into the software folder

STEP 2: Run Scripts in the scripts folder
-Run ./init.sh to setup DV
-Run ./run.sh to run DV. Verify DV started completely with tail -f dv.log
-Run ./test.sh > out.txt to test the OData url

STEP 3: Browse the Data Virtualization and the Data
-View the out.txt to see All Data, Specific Entity and Metadata examples for XML format and/or run the three below individually through the Chrome browser
-All Data
http://localhost:8080/odata/CustomerContextVDB/CustomerContextView.CustomerContextTable?$format=json
-Specific Entity
http://localhost:8080/odata/CustomerContextVDB/CustomerContextTable('123')?$format=json
-Metadata
http://localhost:8080/odata/CustomerContextVDB/$metadata
-Management Console to view Virtual Database
http://localhost:8080
-Dashboard
http://localhost:8080/dashboard/

As Easy as 1,2,3....

OPTION 3 - CONTAINERIZED INSTALL:

STEP 1: Clone the Repository and Download Data Virtualization
-git clone https://github.com/jbossdemocentral/dv-gettingstarted.git
-Download from jboss.org http://www.jboss.org/products/datavirt/download/
Put the Data Virutalization Download, jboss-dv-installer-6.1.0.redhat-3.jar, into the software folder

STEP 2: Create and Run Container

Copy Dockerfile and .dockerignore files from support/docker directory to the project root.

Build demo image

docker build -t jbossdemocentral/dv-gettingstarted .

Start demo container

docker run -it -p 9990 -p 9999:9999 -p 8080:8080 -p 31000:31000 jbossdemocentral/dv-gettingstarted

In many cases, the docker socket may not utilize the localhost interface. Changes may be required to modify the following scripts to utilize the correct interface docker is utilizing
- run.sh

-Run ./test.sh > out.txt to test the OData url

STEP 3: Browse the Data Virtualization and the Data
-View the out.txt to see All Data, Specific Entity and Metadata examples for XML format and/or run the three below individually through the Chrome browser.
-All Data
http://<DOCKER_HOST>:8080/odata/CustomerContextVDB/CustomerContextView.CustomerContextTable?$format=json
-Specific Entity
http://<DOCKER_HOST>:8080/odata/CustomerContextVDB/CustomerContextTable('123')?$format=json
-Metadata
http://<DOCKER_HOST>:8080/odata/CustomerContextVDB/$metadata
-Management Console to view Virtual Database
http://<DOCKER_HOST>:8080
-Dashboard
http://<DOCKER_HOST>:8080/dashboard/

As Easy as 1,2,3....

Teiid and Data Virtualization at the EMEA Partner Conference

2015-08-25T13:00:00.001-04:00

The EMEA Partner conference is October 4th to October 7th, 2015 in Frankfurt at the Congress Center Messe Frankfurt. Data Virtualization is included in 2 presentations and 1 lab.

Presentation 1 - Integration Red Hat Mobile with Data Virtualization

Date/time: Monday, October 5, 2015 - 17:00 to 17:40

Abstract: Being Updated

Presentation 2 - Big Data in Action with Red Hat JBoss Data Virtualization

Date/time: Wednesday, October 7, 2015 - 10:15 to 10:55

Abstract: Hortonworks Data Platform (HDP) with Red Hat JBoss Data Virtualization (DV) integrates Hadoop with existing information sources including data warehouses, SQL databases and enterprise applications. The solution creates business-friendly, reusable and virtual data models with unified views by combining and transforming data from multiple sources including Hadoop so developers and business analysts can build new applications that exploit new data sets. In this session we will discuss HDP and DV integration and show a real world example of Big Data with DV. DV will leverage HDP to mashup Clickstream analysis data with product and customer data on HDP to provide Virtual Data Marts to Marketing and Product Teams for multi-departmental teams.

The Red Hat Partner Conference agenda has a variety session types including keynotes, breakout tracks, Labs Unlimited, examination and more.

Exhibition
Visit the exhibition.
Keynotes
Gain insights from Red Hat and partner executives.
Breakout tracks
Breakout sessions are 40 minute presentations by subject-matter experts focused on a particular topic. Get your existing sales accreditation extended by participating in defined sessions. The defined sessions will be published here.
Labs Unlimited
Over 35 labs, both instructor-led courses (see daily agenda for details) and self paced test labs (ongoing in Spektrum I).
Red Hat Examination*
Red Hat examinations are free of charge training certifications. We offer these examinations during the EMEA Partner Conference:
• Red Hat Certified System Administrator (RHCSA) exam (EX200)
• Red Hat Certified Engineer (RHCE) exam (EX300)
• Red Hat Certified System Administrator in Red Hat OpenStack exam (EX210)
• Red Hat Certified JBoss Developer exam (EX225)
• Red Hat Certified JBoss Administrator (RHCJA) exam (EX248)

JDV and JDG Finalist in the DBTA 2015 Readers Choice Awards

2015-08-11T23:58:00.003-04:00

We are very excited to announce that both JBoss Data Virtualization and JBoss Data Grid were finalist in the Database Trends and Applications (DBTA) Readers Choice Awards for 2015. JBoss Data Grid was a Finalist in the category, Best In-Memory Database. JBoss Data Virtualization was a Finalist in the category, Best Data Virtualization Solution. There were 29 categories in which products could be nominated and ultimately voted upon. The categories were wide-ranging for the Data Industry. For instance, there are Hadoop distributions, NoSQL database products, and cloud products and services, but also categories to recognize products in more established segments of the market such as relational database management, MultiValue, business intelligence, and data integration.

Red Hat JBoss Data Virtualization (JDV) concentrates on three main concepts listed below:

Connect provides Data Federation across one to many disparate data sources so that aggregated views can be built
Compose provides the Data Integration which allows discovery, cleansing, monitoring, transforming and delivery of data from diverse sources
Consume provides Data Abstraction which is the reduction of a particular set of data to a simplified or essential elements of data.

The main concepts bring us to the overall goal of JDV from DBTA which is to achieve a complete view of enterprise data across disparate data environments. As businesses are increasingly using a wide range of technologies for structured and unstructured data that is located across systems there is a risk of moving further from the main goal. But using JBoss Data Virtualization reduces or removes this risk since data from across the enterprise can be abstracted into the essential data services provided through Virtual Databases (VDBs). The data services gives companies the ability to reduce complexity and improve data analysis. A secondary goal is to to allow organizations a complete data view without having to modify or remove physical systems in place.

So to summarize, according to the DBTA, the successful solutions provide agile enterprises agile development, a secure virtual data layer and real-time data access and provisioning.

JDG is an in-memory data grid (IMDG) solution that resides entirely in RAM, and is distributed among multiple servers. Recent advances in 64-bit and multi-core systems have made it practical to store terabytes of data completely in RAM, obviating the need for mass storage media such as hard disks.

According to industry analyst firm Gartner Inc., IMDGs are suited to handle big data's "big-three V's": velocity, variability, and volume. IMDGs can support hundreds of thousands of in-memory data updates per second, and they can be clustered and scaled in ways that support large quantities of data. Specific advantages of IMDG technology include:

Enhanced performance because data can be written to, and read from, memory much faster than is possible with a hard disk.
The data grid can be easily scaled, and upgrades can be easily implemented.
A key/value data structure, rather than a relational structure, provides flexibility for application developers.
The technical advantages provide business benefits in the form of faster decision making, greater productivity, and improved customer service.

Applications that can benefit from IMDG include financial-instrument pricing in banks, shopping carts in e-commerce, user-preference calculations in Web applications, reservation systems in the travel industry, and cloud applications.

So to summarize, the successful solutions provide access to data quickly and easily.

New Data Virtualization Stickers

2015-08-07T10:21:00.000-04:00

We have new Data Virtualization (Teiid) images for stickers available which are great for laptop covers, notebook covers, etc. Watch for these stickers across the globe. I usually order the die cut stickers from Sticker Mule. You can access the Sticker Mule Templates here. You can also get the stickers from the JBoss Middleware Business Unit. Just reach out to our Evangelists, Data Virtualization Product Marketing Manager or Data Virtualization Product Manager.

Enlighten your data

Knowledge is power

One view to query 'em all!

Teiid Designer Diagram Model Size

2015-08-05T22:32:00.001-04:00

Teiid Designer in the JBoss Developer Studio Integration Stack (JBDIS) contains different preferences that allows the Developer to customize their development experience. We will highlight one of the preferences in this article.

Teiid Designer is a visual tool that enables rapid, model-driven definition, integration, management and testing of data services without programming using the Data Virtualization (Teiid) runtime framework. Teiid Designer is a part of the JBoss Developer Studio Integration Stack. The JBoss Tools integration stack is a set of Eclipse plugins that add support for JBoss SOA, Data Virtualization and Business rules related frameworks, such as Drools, Teiid, jbpm, SwitchYard, Fuse, etc.

One of the Teiid Designer preferences is the Diagram Settings. One of the settings is the Model Size. Since displaying very large diagrams may take a considerably long time, this preference allows users to set an upper limit on the number of objects to display in a diagram. If this limit is exceeded, a warning message is displayed to the user and the diagram is not constructed.

To get to this setting click on Window in the menu and then Preferences.

Within the Preferences window, after clicking the Teiid Designer preferences, the preferences will show in the left which include General, Diagram, Editor and Validation settings.

After clicking on the Diagram Preferences, the settings window will display.

The Model Size default is 20000 and can be changed to a larger value.

To find out more about the Teiid Preferences you can take a look at the Product Documentation.

To find out more about how to integrate faster with Red Hat JBoss Middleware and Data Virtualization (DV) you can take a look at the DV Product Pages.

Friday is System Administrator Day

2015-07-30T02:34:00.000-04:00

Happy SysAdmin Day!

Wait… what exactly is SysAdmin Day? Oh, it’s only the single greatest 24 hours on the planet… and pretty much the most important holiday of the year. It’s also the perfect opportunity to pay tribute to the heroic men and women who, come rain or shine, prevent disasters, keep IT secure and put out tech fires left and right.

At this point, you may be thinking, “Great. I get it. My sysadmin is a rock star. But now what?” Glad you asked! Proper observation of SysAdmin Day includes (but is not limited to):

Cake & Ice cream
Pizza
Cards
Gifts
Words of gratitude
Custom t-shirts celebrating the epic greatness of your SysAdmin(s)
Balloons
Streamers
Confetti

Friday, July 31, 2015, is the 16th annual System Administrator Appreciation Day. On this special international day, give your System Administrator something that shows that you truly appreciate their hard work and dedication. (All day Friday, 24 hours, your own local time-zone).

Let’s face it, System Administrators get no respect 364 days a year. This is the day that all fellow System Administrators across the globe, will be showered with expensive sports cars and large piles of cash in appreciation of their diligent work. But seriously, we are asking for a nice token gift and some public acknowledgement. It’s the least you could do.

Consider all the daunting tasks and long hours (weekends too.) Let’s be honest, sometimes we don’t know our System Administrators as well as they know us. Remember this is one day to recognize your System Administrator for their workplace contributions and to promote professional excellence. Thank them for all the things they do for you and your business.

Java EE and Node.js Comparison

2015-07-29T23:55:00.000-04:00

I was looking for a quick comparison of JavaEE and Node.js and came across a blog from StrongLoop that I thought did a good job of a quick comparison including an example for both.

In their post, they showed how to create a REST based web service using Java EE. This service returned a list of beers from the fantastic Pabst Brewing Company. Then created the same application using StrongLoop LoopBack and Node. With very little code in Node, they created a REST web service that returns the name and description of our beers. In addition, the LoopBack API also provides default actions for the entire CRUD process.

The table below shows some of the comparisons discussed in the blog:

Feature	Java EE	Node.js
Great IDE support	Yes, multiple choices including Eclipse, Sublime and Idea	Yes, multiple choices including Visual Studio, Eclipse, Sublime
Dependency management	Maven	NPM
Enterprise ready	Yes, in use today	Yes, in use today
Large ecosystem of libraries	Yes	Yes
Requires JVM	Yes	No
Common frameworks	Spring, JEE	Express
Database support	Yes	Yes
ORM frameworks	Yes	Yes
Testing Frameworks	Yes	Yes

References:

https://strongloop.com/strongblog/node-js-java-getting-started/

http://blog.shinetech.com/2013/10/22/performance-comparison-between-node-js-and-java-ee/

http://www.infoworld.com/article/2883328/java/java-vs-nodejs-an-epic-battle-for-developer-mindshare.html

What is the Hadoop Ecosystem?

2015-07-28T16:30:00.001-04:00

https://www.facebook.com/hadoopers

In some of our Articles and Demos we have examples of JBoss Data Virtualization (Teiid) using Hadoop as a Data Source through Hive. When creating examples of Data Virtualization with Hadoop Environments such as Hortonworks Data Platform, Cloudera Quickstart, etc. there are alot of open source projects included. I wanted to highlight some of those so that you have an overview of the Hadoop Ecosystem. You can find the information below as well as more projects detail in the ecosystem from the hadoop ecosystem table.

Map Reduce - MapReduce is a programming model for processing large data sets with a parallel, distributed algorithm on a cluster. Apache MapReduce was derived from Google MapReduce: Simplified Data Processing on Large Clusters paper. The current Apache MapReduce version is built over Apache YARN Framework. YARN stands for “Yet-Another-Resource-Negotiator”. It is a new framework that facilitates writing arbitrary distributed processing frameworks and applications. YARN’s execution model is more generic than the earlier MapReduce implementation. YARN can run applications that do not follow the MapReduce model, unlike the original Apache Hadoop MapReduce (also called MR1). Hadoop YARN is an attempt to take Apache Hadoop beyond MapReduce for data-processing.
HDFS - The Hadoop Distributed File System (HDFS) offers a way to store large files across multiple machines. Hadoop and HDFS was derived from Google File System (GFS) paper. Prior to Hadoop 2.0.0, the NameNode was a single point of failure (SPOF) in an HDFS cluster. With Zookeeper the HDFS High Availability feature addresses this problem by providing the option of running two redundant NameNodes in the same cluster in an Active/Passive configuration with a hot standby.
HBase - Google BigTable Inspired. Non-relational distributed database. Ramdom, real-time r/w operations in column-oriented very large tables (BDDB: Big Data Data Base). It’s the backing system for MR jobs outputs. It’s the Hadoop database. It’s for backing Hadoop MapReduce jobs with Apache HBase tables.
Hive - Data Warehouse infrastructure developed by Facebook. Data summarization, query, and analysis. It’s provides SQL-like language (not SQL92 compliant): HiveQL.
Pig - Pig provides an engine for executing data flows in parallel on Hadoop. It includes a language, Pig Latin, for expressing these data flows. Pig Latin includes operators for many of the traditional data operations (join, sort, filter, etc.), as well as the ability for users to develop their own functions for reading, processing, and writing data. Pig runs on Hadoop. It makes use of both the Hadoop Distributed File System, HDFS, and Hadoop’s processing system, MapReduce. Pig uses MapReduce to execute all of its data processing. It compiles the Pig Latin scripts that users write into a series of one or more MapReduce jobs that it then executes. Pig Latin looks different from many of the programming languages you have seen. There are no if statements or for loops in Pig Latin. This is because traditional procedural and object-oriented programming languages describe control flow, and data flow is a side effect of the program. Pig Latin instead focuses on data flow.
Zookeeper - It’s a coordination service that gives you the tools you need to write correct distributed applications. ZooKeeper was developed at Yahoo! Research. Several Hadoop projects are already using ZooKeeper to coordinate the cluster and provide highly-available distributed services. Perhaps most famous of those are Apache HBase, Storm, Kafka. ZooKeeper is an application library with two principal implementations of the APIs—Java and C—and a service component implemented in Java that runs on an ensemble of dedicated servers. Zookeeper is for building distributed systems, simplifies the development process, making it more agile and enabling more robust implementations. Back in 2006, Google published a paper on "Chubby", a distributed lock service which gained wide adoption within their data centers. Zookeeper, not surprisingly, is a close clone of Chubby designed to fulfill many of the same roles for HDFS and other Hadoop infrastructure.
Mahout - Machine learning library and math library, on top of MapReduce.

Also, you can visit the Big Data Insights Page to learn more about Red Hat Products in relation to the Hadoop Ecosystem.

Suggested Minimum Requirements for Data Virtualization Server and Designer

2015-07-27T21:46:00.000-04:00

This week I wanted to share some minimum requirement recommendations for Data Virtualization (Teiid) Server and Designer which are below. You can find more detail in the Product Documentation such as the three considerations that help determine the minimal JVM footprint - concurrency, data volume and plan processing. Also you can find more detailed environment recommendations through the sizing tool. The JBoss Data Virtualization Sizing Architecture Tool is a simple web application that helps plan your JBoss Data Virtualization deployment. It presents a series of questions to gather information about the business environment for the deployment. When all questions are answered, the tool recommends a JBoss Data Virtualization configuration to support the business requirement. This recommendation includes:

The recommended number of servers
The recommended number of cores
How much memory is required, and the JVM size for needed, for each node

The following recommended minimum requirements should be thought of as a starting point. These should be adjusted from the additional suggestions from the formulas in the product documentation, sizing tool and based on expected usage.

JBoss Developer Studio Integration Stack (Teiid Designer) – without application server

2 GB RAM will get you started, but more is needed for large models
Modern Processor
500 MB disk space for installed product files
2+GB for model projects and related artifacts

The minimum sizing for the DV server is:

16 GB JVM memory size
Modern multi-core (dual or better) processor or multi-socket system with modern multi-core processors
20+ GB Disk Space that's needed for JBoss server product and DV components:
1 GB disk for installed product files
5+ GB for log files and deployed artifacts
15 GB (default) for BufferManager maxBufferSpace
If Modeshape (repository) will be used, will need to bump up the file space by a minimum 5GB more.

Connecting to Cloudera Quickstart Virtual Machine from Data Virtualization and SQuirreL

2015-07-23T02:46:00.000-04:00

http://www.redhat.com/en/files/resources/en-rhjb-ventana-research-infographic.pdf

One of the great capabilities of JBoss Data Virtualization is the ability to connect to Hadoop through Hive which was added as part of Data Virtualization (DV) 6.0. This gives us the ability to aggregate data from multiple datasources that include big data. This also gives us the ability to analyze our data through many different tools and standards. From the Re-Think Data Integration infographic, more than one-quarter of companies see virtualizing data as a critical approach to integrating big data. With DV 6.1 Cloudera Impala was added for fast SQL query access to data stored in Hadoop. So I wanted to add an example of how to use Cloudera Impala as a Data Source for DV. To find out how Cloudera Impala fits into the Hadoop ecosystem take a look at the Impala Concepts and Architecture documentation.

Differences between Hive and Impala

First let's take a look at an overview of Hive and Impala. Cloudera Impala is a native Massive Parallel Processing (MPP) query engine which enables users to perform interactive analysis of data stored in HBase or HDFS. The Apache Hive data warehouse software facilitates querying and managing large data sets residing in distributed storage and provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL. I grabbed some of the information from the Cloudera Impala FAQ.

How does Impala compare to Hive (and Pig)? Impala is different from Hive because it uses its own daemons that are spread across the cluster of queries. Since Impala does not rely on MapReduce it avoices the startup overhead of MapReduce jobs which allows Impala to return results in real time.

Can any Impala query also be executed in Hive? Yes. There are some minor differences in how some queries are handled, but Impala queries can also be completed in Hive. Impala SQL is a subset of HiveQL. Impala is maintained by Cloudera while Hive is maintained by Apache.

Can I use Impala to query data already loaded into Hive and HBase? There are no additional steps to allow Impala to query tables managed by Hive, whether they are stored in HDFS or HBase. Impala is configured to access the Hive metastore.

Is Hive an Impala requirement? The Hive metastore service is a requirement. Impala shares the same metastore database as Hive, allowing Impala and Hive to access the same tables transparently. Hive itself is optional.

What are good use cases for Impala as opposed to Hive or MapReduce? Impala is well-suited to executing SQL queries for interactive exploratory analytics on large data sets. Hive and MapReduce are appropriate for very long running batch-oriented tasks such as ETL.

Cloudera Setup

There are several options that Cloudera offers to test their product. There are quickstart VMs, Cloudera Live and Local install for CDH. For quick setup I chose a quick start Virtual Box Virtual Machine for CDH 5.4.x for a single-node Hadoop Cluster with examples for easy learning. The VMs run CentOS 6.4 and are available for VMWare, VirtualBox and KVM. All of them require 64-bit host OS. I also chose a bridged network, increased the memory and CPUs. Once the VM is downloaded you extract the files from the ZIP, import the VM and make network/memory/CPU setting changes. Then the VM can be started.

Once you launch the VM, you are automatically logged in as the cloudera user. The account details are:

username: cloudera
password: cloudera

The cloudera account has sudo privileges in the VM. The root account password is cloudera. The root MySQL password (and the password for other MySQL user accounts) is also cloudera. Hue and Cloudera Manager use the same credentials. Then we want to browse to the Cloudera Manager at quickstart.cloudera:7180/cmf/home. We want to make sure our services are started such as Impala, Hive, YARN, etc.

Now that we have the VM setup we want to make sure we can add data. I ran through Cloudera's Tutorial Exercise 1-3 at quickstart.cloudera/tutorial/home. Then we can see the tables and data through the Hue UI (quickstart.cloudera:8888) through impala and hive.

Now we have our Big Data Environment running so let's move onto testing Impala and Hive.

SQuirreL Testing

SQuirreL SQL Client is a free open source graphical SQL client written in Java that will allow you to view the structure of a JDBC compliant database, browse the data in tables, issue SQL commands etc. First we have to download the JDBC drivers for Impala and Hive. First we will go through Impala in SQuirrel. I downloaded the Impala JDBC v2.5.22 driver. Then I unzipped the jdbc4 zip file. We add the JDBC driver by clicking on the drivers tab, then the add driver button. In the extra class path tab click on Add, browse to the extracted jar files and add them along with the Name and Class Name.

Once the driver is added the driver should have a green check mark.

Next we click on the Aliases tab and add a new connection.

We add the driver and URL and then click connect. Once connected we can browse to the tables, select a table and preview content.

Now we have previewed the data through Impala through SQuirreL. Now we want to test Hive as well. We download the Hive 1.2.1 driver from Hive Apache. We do the same as above and add the driver by pointing to the jars in the lib directory of the download and use Driver Class org.apache.hive.jdbc.HiveDriver. Once the driver is added then we create a session connecting to Hive.

We can view the tables and content which shows we can now use Hive and Impala through DV to access the data in Cloudera.

Data Virtualization Testing

Now we can move onto testing Cloudera Impala with DV. DV 6.1 GA can be downloaded from jboss.org. We will also use JBoss Developer Studio and the JBoss Integration Stack.

Data Virtualization Setup

Run through the DV install instructions to install the product. Then we want to update the files for the Cloudera Impala JDBC driver.

To configure the Cloudera's Impala as datasource with DV

1) Make sure the server is not running.
2) In the modules directory create the org/apache/hadoop/impala/main folder.

3) Within the folder we want to create the module.xml file.

<?xml version="1.0" encoding="UTF-8"?>
<module xmlns="urn:jboss:module:1.0" name="org.apache.hadoop.impala">
    <resources>
      <resource-root path="ImpalaJDBC4.jar"/>
      <resource-root path="hive_metastore.jar"/>
      <resource-root path="hive_service.jar"/>
      <resource-root path="libfb303-0.9.0.jar"/>
      <resource-root path="libthrift-0.9.0.jar"/>
      <resource-root path="TCLIServiceClient.jar"/> 
      <resource-root path="ql.jar"/>        
    </resources>

    <dependencies>
        <module name="org.apache.log4j"/>
        <module name="org.slf4j"/>
 <module name="org.apache.commons.logging"/>
        <module name="javax.api"/>
        <module name="javax.resource.api"/>        
    </dependencies>
</module>

Note that the resources points to the jar files included with the Impala driver.

4) Now copy the above JDBC jar files in the resources section to the folder.
5) Next we update the standalone.xml file in the standalone/configuration folder. The driver and datasource can be added.

<datasource jndi-name="java:/impala-ds" pool-name="ImpalaDS" enabled="true" use-java-context="true">
     <connection-url>jdbc:impala://10.1.10.168:21050/;auth=noSasl</connection-url>
     <driver>impala</driver>
</datasource>

<driver name="impala" module="org.apache.hadoop.impala">
     <driver-class>com.cloudera.impala.jdbc4.Driver</driver-class>
</driver>

6) Now we can start the server

JBoss Developer Studio Setup

Run through the install instructions for installing the JBoss Developer Studio and installing the Teiid Components from the Integration Stack. Create a new workspace, ie dvdemo-cloudera. Then we will create a example view.

1. Create a New Teiid Model Project, ie clouderaimpalatest

2. Next Import metadata using JDBC from a database into a new or existing relational model using Data Tools JDBC data source connection profile.

3. Next we create a new connection profile with the Generic JDBC Profile Type.

4. We create a new driver, ie Impala Driver, by adding all the jars and setting the connection settings. The username/password should be ignored.

5. Next we import the metadata and select the types of objects in the database to import. We will just choose the tables and then the sources folder.

6. We add a new server and set it as externally managed. We start the server externally and then click the start button within JBDS,

7. Within the sources folder in the Teiid Perspective we right click one of the tables then Modeling and Preview Data. If everything is setup properly then Data will display.

8. Now we can create a layer or View to add an abstract layer. In our case we are just going to create a View for a one to one to Source example. But to show the power of DV we would normally aggregate or federate multiple sources into a view either in this layer or create another layer above that uses the lower layers for greater abstraction and flexibility. We will test with the customers table. After creating our view with table we tie the source table to the view. We also set the primary key on the customerid so when we create the VDB OData is available. We can also preview the data on the view.

9. We create a VDB that we can deploy and execute to the server.

10. After right clicking on the clouderaimpalatest.vdb we click on deploy so it is deployed to the server. Next we can browse to the OData to show the data as a consumer.

-First we take a look at the metadata

-Then we can list all the customers

References

https://hive.apache.org/
https://developer.jboss.org/wiki/ConnectToAHadoopSourceUsingHive2

http://www.cloudera.com/content/cloudera/en/downloads/connectors/impala/jdbc/impala-jdbc-v2-5-5.html

https://github.com/onefoursix/Cloudera-Impala-JDBC-Example

Tested Integrations for Data Virtualization

2015-07-21T18:53:00.000-04:00

Every JBoss® Data Virtualization (JDV) release is tested and supported on a variety of market-leading operating systems, Java™ Virtual Machines (JVMs), and database combinations. In addition, Red Hat provides a tool to help plan your JBoss Data Virtualization deployment JBoss Data Virtualization Sizing Architecture Tool

The following databases and database drivers were tested as part of the JBoss Data Virtualization release process for use as a persistence store for platform components. A persistence store is used by product components to store state or other information. Product components include the ModeShape hierarchical database, S-RAMP repository and Teiid/JDV external materialized views.

Databases	JDBC Driver
IBM DB2 9.7	IBM DB2 JDBC Universal Driver Architecture 4
Oracle 12c	Oracle JDBC Driver v12
Oracle 11g	Oracle JDBC Driver v11
MySQL 5.5	MySQL Connector/J 5
Microsoft SQL Server 2012	Microsoft SQL Server JDBC Driver 4
Microsoft SQL Server 2008 R2 SP2	Microsoft SQL Server JDBC Driver 4
PostgreSQL 9.2	JDBC4 Postgresql Driver, Version 9

The following were tested as part of the JBoss Data Virtualization release process as data sources that may be accessed through the platform.

Data source	Database Driver/Supported Interface/Notes
Oracle 12c	Oracle JDBC Driver v12
Oracle 11g R2	Oracle JDBC Driver v11
Oracle 10g R2	Oracle JDBC Driver v10
IBM DB2 9.7	IBM DB2 JDBC Universal Driver Architecture v4
Microsoft SQL Server 2008, 2012	Microsoft SQL Server JDBC Driver 4
Sybase ASE 15	jConnect for JDBC3.0 v7
MySQL 5.5	mysql-connector-java v5.5
MySQL 5.1	mysql-connector-java v5.1
PostgreSQL 9.2	JDBC4 PostgreSQL Driver, Version 9.2
PostgreSQL 8.4	JDBC4 PostgreSQL Driver, Version 8.3
Teradata 12	Teradata JDBC driver, v13.00
Netezza 6.0.2	Netezza JDBC driver, v6
Greenplum Database 4.1	JDBC3 PostgreSQL Driver, Version 8.3
Ingres 10	Ingres JDBC driver, v3.0
MongoDB v2.4	via Teiid MongoDB resource adapter
Apache Cassandra 2.1 (technical preview)	via Teiid Cassandra resource adapter
JBoss Data Grid 6.4	via Data Grid HotRod client
JBoss Data Services Platform 5.x	JBoss Data Services Platform (Teiid) JDBC Driver
MetaMatrix 5.5.4	MetaMatrix 5.5.4 JDBC Driver
LDAP 3	via Teiid LDAP resource adapter
Salesforce.com	API version 22
SAP NetWeaver Gateway	OData + SAP Annotations for OData v2.0
ModeShape Hierarchical Database	via bundled ModeShape JDBC driver
Files	Delimited and Fixed-length
XML Files
XML over HTTP	HTTP 1.1
SOAP Web Services	SOAP 1.1, WSDL 1.2
Cloudera CDH 5	v2.1 Impala JDBC Driver v0.13, 0.14 Hive JDBC Driver
Hortonworks Data Platform 2	v0.13, 0.14 Hive JDBC driver
Apache Hive 0.12, 0.13, 0.14	v0.12, 0.13, 0.14 Hive JDBC driver
Google Spreadsheets	via Teiid Google resource adapter
Microsoft Excel 2010	Apache POI
Microsoft Access 2010	JDBC-ODBC bridge driver

Node Package Manager Quick Overview

2015-07-14T00:16:00.005-04:00

npm stands for Node Package Manager. The most common use case for npm is to use it for installing modules from other people. Also, using it in development, makes it easier to specify and link dependencies. npm is bundled and installed automatically with the Node.js environment. npm runs through the command line. It allows users to install Node.js applications that are available on the npm registry. npm is written entirely in JavaScript and was developed by Isaac Z. Schlueter. Also npm packages can be installed Globally or Locally. A good rule of thumb, Global packages are for anything that you need to access from the shell. By contrast local packages are for using within your apps. The video below gives a quick overview of npm.

https://docs.npmjs.com/