Redshift challenge to load 1M rows via ODI/JDBC WITHOUT S3 buckets | LIVE at #OOW14

Redshift challenge to load 1M rows via ODI/JDBC WITHOUT S3 buckets | LIVE at #OOW14

Posted on September 15, 2014 0 Comments

Welcome to the Amazon Redshift insert challenge (no s3 or ice buckets).  So LIVE at Oracle OpenWorld 2014 #OOW14, I have been challenged by data professionals to load 1 million records into Amazon Redshift from Oracle Data Integrator WITHOUT using an S3 bucket during a live #OOW14 session: Oracle Data Integration: A Crucial Ingredient for Cloud Integration [CON7926]:

Beyond Redshift | Oracle and Cloud Data Integration:

Amazon Redshift is just one cloud data source, so we'll also demo and talk about connecting various cloud data sources to build an Oracle data warehouse with ODI listed below.  We know this is a hot topic since my last two sessions on Salesforce data integration at Dreamforce 12 and 13 both sold out.

  • Oracle Marketing Cloud (Eloqua)
  • Oracle Service Cloud (RightNow)
  • Salesforce.com
  • Marketo
  • Microsoft Dynamics
  • Google Analytics / Adwords

Challenge not looking good so far ...

  • ODI Job: Loading 1M records into the supplier sample table in Amazon Redshift
  • Work tables are created in Redshift target for ELT
  • Source data is in an Oracle 11g Database
  • Data Integration platform is Oracle Data Integrator 12c
  • Open source 8.4 PostgreSQL JDBC driver
  • Everything is running on a local OracleVM

Since 1 million rows takes awhile, I put a filter to only load 40,288 rows which is taking 790 seconds or ~13 minutes.  We're looking at close to 6 hours to load 1M at this rate. odi-redshift1

odi-redshift2

 

How do I get myself out of this?

  • Is it even possible to get this load within 10 minutes?
  • Moscone may have flaky wifi?
  • Laptop could be confiscated by TSA!
  • May fall off the GG bridge during the 5K OpenWorld Bridge Run Sunday.

But I'm not worried about it since I am backed by the greatest R&D team for data connectivity in the world at Progress DataDirect; and plan to use the new DataDirect Amazon Redshift JDBC Driver.

If we pull this off, I will ask Julien Testut, Principal Product Manager for ODI, who is hosting the session to buy everyone in the audience a local treat (Morning Bun or Cup of Philz?).

Learn more about how DataDirect makes the impossible possible in data connectivity

Get started with a trial download today.

Or tweet to @DataDirect_News, talk to us on live chat or call 1-800-876-3101.

Sumit Sakar

Sumit Sarkar

Technology researcher, thought leader and speaker working to enable enterprises to rapidly adopt new technologies that are adaptive, connected and cognitive. Sumit has been working in the data access infrastructure field for over 10 years servicing web/mobile developers, data engineers and data scientists. His primary areas of focus include cross platform app development, serverless architectures, and hybrid enterprise data management that supports open standards such as ODBC, JDBC, ADO.NET, GraphQL, OData/REST. He has presented dozens of technology sessions at conferences such as Dreamforce, Oracle OpenWorld, Strata Hadoop World, API World, Microstrategy World, MongoDB World, etc.

Comments

Comments are disabled in preview mode.
Topics

Sitefinity Training and Certification Now Available.

Let our experts teach you how to use Sitefinity's best-in-class features to deliver compelling digital experiences.

Learn More
Latest Stories
in Your Inbox

Subscribe to get all the news, info and tutorials you need to build better business apps and sites

Loading animation