data warehouse modernization

Modernize your data warehouse to deal with big data

In today’s digital data landscape, we are generating and accumulating huge variety, volume, and velocity of data. This is a major success in terms of insights which we can derive from the data to assist decision making. However, many organizations are still facing challenges in implementing a standardised process to store, prepare and analyze these data sets.

The solution is to seek a smarter Big Data management solution.

Data Warehouse modernization: A simple solution to big data management

In order to deal with these newer, high volume datasets, organizations are implementing enterprise-grade data lakes on advanced big data technologies like Hadoop, Amazon Redshift and Google BigQuery. These data lakes are used as parallel storage and processing platform to the existing data warehouse systems. With the new age advanced analytics tools, users can integrate and analyze their existing data warehouse with the new data lakes.

Preventing data overload

Another situation where data lakes can come to rescue when organizations are faced with either of the below situations:
Company’s’ current data warehouse cannot scale to support the amount of data that is being recorded
Handling unstructured data received from sources like social media, machine logs, sensors, web sources etc., which are cannot be handled by our underlying data warehouse.
The associated cost of storing and maintaining such datasets in existing warehouse system is high

Data lakes pave the path for unrestricted analytics and helps in capturing information which was not possible earlier due to data warehouse restrictions.

When used and applied correctly, organizations can see 3 key benefits:

  • Data at your finger tips: Data lake makes current and historical data available for running analysis and thereby enabling business users to make more informative decisions.
  • Centralized storage to all of your data: Analysts can integrate data coming from multiple data sources and of different structures and include them in their analysis. They can build correlations and patterns and derive deep dive insights to get a consolidated view of their enterprise.
  • Faster analytics: The newer big data technologies are optimized for parallel processing and faster query response times on even peta-byte scale data. Compared to traditional data warehouses with which queries would run into hours and days, big data technologies ensure an in-time query performance.
data warehouse modernization pipeline


Finding the right solution

Migrating data from a data warehouse to big data platform is easier said than done. It can very well be extremely expensive and time consuming, depending upon the technology you choose.

Following are the three data lake platforms that are seeing good traction currently:

Apache Spark and Hadoop

Apache Hadoop is an open source framework that excels in distributed storage and processing of big data with the ability to scale up to several petabytes of data. Apache spark processes data in-memory and enables batch, real time and advanced analytics on top of Hadoop. With a combination of Hadoop and Spark, organizations can store all-structure data, build data pipelines and analyze data at scale.

Amazon Redshift

Redshift being a fully managed data warehouse solution allows users to run queries in sub-seconds to seconds of latency on their big data. Modern analytics tools can connect directly to Redshift and connect directly to Redshift.

Google BigQuery

The increase in the amount of data organizations are capturing and processing will continue to grow in coming years. To get the maximum benefits. Choosing the right big data strategy to get the most from your data is now critical.

Self Service Business Intelligence and Data Preparation tool for big data on AWS


Enable interactive data exploration and visual analytics of disparate sources on AWS

We are happy to announce that Ideata Analytics, a big data intelligence platform is now available on the AWS Marketplace. It provides a turn-key preconfigured analytics solution on the cloud. Connecting to various data sources, cleaning it and doing ad-hoc analysis to create insights and visualizations has never been easier and faster thanks to Ideata Analytics cloud based analytics platform.

You can get a free 15 day trial by visiting the AWS Marketplace or going directly to

It provides out of the box connectors to major AWS services. The extensive list of data connectors includes support for Amazon Redshift, AWS S3, AWS EMR, Amazon aurora. Apart from that it also supports connectors to various big data sources like Hadoop, Spark, MongoDB, traditional RDBMS like mysql, oracle and mssql and files like csv,excel,json, xml and more.

This will really be useful for users who wants to leverage their existing AWS cloud infrastructure and want to combine it with the performance and scalability of a modern big data solution. It will be very simple to use, fully managed and invoiced through a single AWS account” said Pranjal Jain, founder of Ideata Analytics. “One can easily spin up an Ideata Analytics machine from marketplace and connect, clean and analyze their data kept in AWS sources or other external databases and files. It will help them jumpstart their analytics cycle within minutes on cloud

Ideata Analytics is built from ground up using latest big data technologies like apache spark, which it uses as its core processing engine. It empowers users to work with billions of data points and apply live transformation on it. The interactive and intuitive platform is designed for business users so that they can drag and drop data columns to build visualization and drill down and drill across datasets to reach to the point of interest.

The platform also speeds access, processing, and analysis of data on Amazon Redshift. It provides fast interactive analysis and data discovery capabilities to uncover hidden insights and a way for users to blend disparate data on the fly.

About Ideata, Inc.

Ideata Inc is committed to work with businesses to enable faster, better decision making using it’s end-to-end data analytics platform. Ideata Inc team includes veterans from banking and telecommunication industries with years of experience in big data, BI and analytics. The company has partnered with companies including Cloudera, Hortonworks, MapR, HP, IBM and AWS.

Test drive Ideata Analytics today – Free 15 days trial !

The Modern Analytics Approach Marketers Should Use

In the digital marketing era, every marketer has access to systems and processes to provide them all the data related to their marketing spends and efforts. The challenge, however, is to derive actionable insights from this enormous amount of data.

Challenges in getting to insights


The usual approach which they take is to hire more and more data scientists and engineers which can provide them with the insights they are looking for. But, due to the shortage of time and skilled force, the overall cost, time and associated effort balloons up.

To get a holistic picture marketers should look into all available data sets like weblogs, call center transactions, online visitors clickstreams etc to better understand their customers and plan appropriate strategy. However integrating and leveraging these datasets in their analysis is a complex and time-consuming task and often requires technical expertise.


The modern approach



To make a scalable and cost effective process, organizations need to empower existing marketing workforce with a self service analytics tool.

These tools are designed in a way that even non techincal users can perform self service data analytics and discovery on their data themselves. These user-friendly and interactive tools provide easy to use drag and drop interface which enables marketers to reach to insights without writing any code. They can access the entire data and use it to make decisions. Self service analytical tools give marketers the much-needed flexibility and ability to define rules on the fly.

It helps them move away from the rigid processes and adapt to the changing market conditions and react faster to customer interests. They can spot trends, build correlations and identify anomalies.

As the owner and the knower of the overall marketing strategy, marketers are the best person to ask the right questions. They know the processes, domain and data, they know which KPI will work. Using these tools, they can quickly test their hypothesis and optimize their campaign with data driven facts in no time.


With a smart and well-designed analytics process, non-technical marketers can also quickly integrate and merge data from different data silos to build converged insights. This new breed of analytics tools will empower marketers to question data and detect patterns whenever they want at their fingertips.

How to visualize your sales data on a world map


In this tutorial, we will quickly see how we can use Ideata analytics to quickly plot sales number on an interactive world map to get a detailed perspective of your sales by geography.

Step 1- Connect to data

We have sample sales data in our local MySQL database. In order to start visualizing our data, we will first connect to MySQL and import relevant tables.


Step 2- Enrich data with location specific details


Once the data is loaded the application will display a preview of the data. We can see that there are columns in the data which show us the store address. We will split the address column to extract city name and state.

In order to plot these locations on the world map, we will need actual latitude and longitude of the store. We can use the inbuilt geo co-ordinate lookup to perform this operation. From the drop-down menu in the newly created city column we will select “Lookup geo coordinates for location”.



This will generate latitude and longitude against all the city names.

We will go ahead and click on Finish button on the bottom right corner and import our data.

Step 3 — Plot your sales number on World Map.


Once the data is imported, we will click on the newly created data source which will take us to the analysis page. We will select the chart type as the map from the chart dropdown. From chart option panel on the right, we will select the map type as a pie chart.

From the left panel, we will drag and drop latitude and longitude columns in the top bar. This will plot the values on the map. We will go ahead and drag sales amount and product category on to “SIZE” and “GROUP BY” fields on the bottom left panel.



This will show us the detailed map marked with store locations. The size of the circles and numbers in them shows us the total sales and the different colors of the pie depicting sales for each product category in the given region. You can zoom into any of these locations to get a more granular view of the data.

Try it out on your own data by registering for a trial. Click here: