Rise of the Data Engineer
The life of an average Indian student is caught in the web of traditional engineering. In the anticipation of building a comfortable life, all hopes are placed on an engineering degree. So much so that an engineer in the family is like an insurance against life’s oddities.
We cannot refute that this held true for more than a couple of decades. Engineering colleges in the country blossomed like mushrooms. Companies hired engineers in bulk. The Indian middle class thrived. Education loans multiplied. We saw migration of engineers to the United States, the Silicon Valley.
Everything was going well and then in the mid-2017, the news journal Mint reported that the top seven IT firms in India including Infosys, Wipro and Cognizant were all set to lay off at least 56,000 employees by the end of the year.
Online forums were abuzz with speculation about the job market and the impending crisis. Quora and Facebook groups received distressed questions such as “How is automation affecting the layoffs in the IT industry?” and “Is Cognizant laying off 6000 people? Etc.
Is this an engineering bubble which has already burst? In a study conducted by The Hindustan Times, it was found that only 66% of the IIT Graduates found campus placements. Out of which only very few found jobs relating to their core competency. Economic slowdown is not only to be blamed.
Interestingly, the study found that there is no dearth of jobs. But, even the premier institutes are not able to match the demand for the skill set required by the rapidly changing industry. What the students lack is modern infrastructure, quality faculty, internship, live training, and regular upgradation of skills.
Now let us come to Data Engineering. Data Science is the new engine that drives the world and Data Engineers are the skilled workers who make this drive smooth and possible.
Information is power, and Data Science Engineers use their skills to dig information out of mountains of data.
Data Engineers are software engineers who design, build, integrate data from various resources, and manage big data. They write complex queries on data and make sure that it is easily accessible for analysis. Their goal is to optimize the performance of their company’s big data ecosystem.
Data engineers will increasingly be an integral part of the movement to use data for making better business decisions. Between 2013 and 2015, the number of data engineers more than doubled. And as of January 2018, there were over 2,800 open positions with “data engineer” in the title on LinkedIn, indicating the growing demand for this specialty.
A bird’s eye view of the Data Engineering Landscape
As seen in the illustration above, the naming of the job roles may be different, but essentially all of it comes under the gamut of Data Science Engineering Professionals (DSEP).
The DSEP, make sure that the data is collected from various sources, converted to a legible form and are made available to the Data Scientists to perform advanced analytics. The range of their responsibilities varies from data munging, data quality governance to basic visualization.
A thirst for knowledge and constant eye on the changing technology trends is the only insurance that an engineer can depend on, to safeguard his interests.
Data Engineering typically requires a basic engineering degree with specialized Data Science skills. Backgrounds are less strictly related to computer science. Mathematicians, statisticians, physicists, and econometricians also often make great data engineers.
Engineers are very apt to make a career in data Science given their solid background in technology & mathematics. Data engineers make sure that reliable and timely information is available for the Data Scientists to build a story on. As we move towards the “Data Era”, the role of the data Engineer becomes extremely important as a bridge between the DBA & the end data user.
Given that the data revolution continues, Data Engineers will be in high demand and the pay scale corresponds to the demand.
If you love playing with new tools and can think outside the relational database box, you’ll be in a prime position to help companies adapt to the 21st century.
