Since the onset of the pandemic, IT has risen in prominence as an engine for business sustainability and growth across all industries. The subsequent demand for enterprise IT talent has led to a sharp spike in salaries CIOs must pay to staff their teams.

“Demand for tech talent was up by 50% to 60% in the last two years, mainly attributable to industry-spanning hyper digitalization. Companies across verticals were ramping up digital functions with the increasing demand for innovation in products and service offerings, especially since the onset of the pandemic,” says Sachin Alug, CEO of talent solutions company NLB Services. “As companies strive to transform their business models specifically in manufacturing, supply chain, project management, and sales, the uptick in hiring IT talent was seen as a natural progression.”

While the rise in salaries for technology roles has bode well for IT professionals, it has posed challenges for CIOs in effectively managing their budgets with the increased demand on their IT departments.

“In response to the demands from business, the strength of our IT department has gone up from 20 to 50 in the last two years,” says Ranganathan Iyer, CIO at auto components manufacturer JBM Group. “The new resources have come at a 30% to 40% additional cost but there has only been an incremental y-o-y increase of 10% to 12% in our budgets.”

Iyer was expecting the massive layoffs taking place in the US to reflect in India too, which could have eased salaries. But while Big Tech in the US has retrenched thousands of workers since the pandemic as recession looms large over its economy, that hasn’t happened in India.

Describing India as a “bright spot” in the global economy, Kristalina Georgieva, MD of International Monetary Fund, has said the country will contribute 15% of the global growth in 2023. According to recruitment and staffing services company Randstad, Indian technology firms are expected to buck the global trend and start hiring in 2023. It is this Indian growth story that is causing high salaries to hold up, at least for the time being.

“Today we are getting resources with half the experience at the same salary as an employee with over 20 years of experience in the company. In such a scenario, it is tough to retain senior employees,” says Iyer, who has no choice but to hire when it comes to next-generation technologies such as AI and ML. “We can upskill existing resources in other areas but not in these.”

Iyer is finding it tough to get cost-effective resources even though he doesn’t hire from top institutes such as IITs (Indian Institute of Technology) and NITs (National Institute of Technology).

“We expect the situation to ease by October as the impact of the global layoffs will be felt by then. Indians being retrenched in the US could return home. But the new benchmark in salaries has already been set. It is 30% higher than pre-COVID levels,” says a senior technology leader from the travel industry, on condition of anonymity.

A top IT decision maker from the BFSI vertical is feeling the heat too. “Our outsourcing partners faced a lot of issues in the last two years. There was high demand for coders, and skilled resources left them at 100% to 120% salary hikes. Some of our contracts with them, which were very old, exhausted in months because of churn in these vendors,” he says, not wanting to be quoted.

With the situation reaching a climax, CIOs have no choice but to find innovative solutions to tackle the situation.

Managing talent costs through outsourcing

To control costs, most CIOs are taking recourse in outsourcing. As the CIO from the BFSI vertical says, “There was a 30% increase in salaries, but it was absorbable as we had already outsourced most applications such as mobile banking and internet banking. The freed resources were diverted to other areas such as infosec, VPN, and networking. They are also being upskilled on next-gen technologies as we look to set up a center of excellence in a specific technology.”

Similarly, the technology leader from the travel industry has outsourced both the company’s core IT infrastructure and its security operations center. However, he kept other areas, such as DevOps, inhouse. “Although we have hired 50%-60% more resources, our overall budget hasn’t exceeded. As a result of outsourcing, we haven’t disturbed the balance sheet. We have also entered long-term contracting to insulate ourselves from price rise,” he says.

Iyer has already outsourced 90% of the company’s IT — everything from the network to the hardware maintenance. Only the ERP and the AI divisions are inhouse. About five years back he thought of outsourcing the ERP division too but then realized it would end up costing more. “The outsourcing partner would only take care of the maintenance. For any new testing, development of business, they would charge extra. We also thought of setting up a shared SAP competency center with other big auto ancillary players, but it didn’t materialize,” he says.

Embracing skills-based hiring and upskilling

Eliminating role-based hiring and making skill-based hiring the mandate can be another approach that CIOs across verticals can adopt amid turbulent times.

“As technology continues to evolve and change the business landscape, the focus needs to be shifted to expertise rather than experience when it comes to hiring IT resources,” says Alug, of NLB Services. “Studies have shown that employers today are more inclined to hire skilled candidates rather than experienced candidates. Many high-tech organizations have resorted to hiring freshers in numbers higher than experienced professionals as the former comes with updated domain knowledge in the emerging technologies.”

Iyer too is banking on upskilling as he looks to implement SAP HANA by the end of the year. He is planning to upgrade the skillsets of resources within the company instead of hiring expensive ones from outside.

“Today, the list of IT and non-IT companies axing redundant roles is growing. In such a situation, employees must keep their skills in check to prove their indispensability within the organization. When it comes to tech and digital expertise, the most impactful means to raise productivity in tandem with the evolving technology is constant upskilling,” says Alug. “In a digital world, the list of certifications for IT courses is endless. However, there is an increased demand for skill building in certain areas such as cloud, data science, DevOps, AI/ML, and cybersecurity to keep up with the progressing technologies.”

Shifting IT strategies to curb costs

In another initiative, the technology leader from the travel industry has “reengineered his enterprise’s operations.”

“The travel industry has lots of legacy applications and APIs. We looked at those applications that were non-revenue generating and shut them down. Overall, 30% of the applications were done away with. With all these initiatives, we are today working at 60% less operating cost,” he says.

The CIO from the BFSI vertical says his enterprise kept costs down by shutting 30% of its regional offices and persisting the company’s work-from-home policy.

Meanwhile, his outsourcing partners have adopted an innovative approach to manage the situation. “Rather than laying off employees, their salaries have been cut and they have been given ESOPs [Employee Stock Ownership Plans]. As the company’s P&L moves, so would their benefits,” he says.

Vendors are also offering innovative schemes to help CIOs manage costs better. “Vendors such as Dell and HP have launched an innovative plan. Whenever a new employee joins, we had to give them new laptops. Now these vendors take back old laptops and return after refurbishing them by replacing their old parts such as keys and screen. This has extended the life of a laptop from 18 to 20 months to 64 to 70 months, helping us save 40% costs on a month-on-month basis,” says the BFSI IT leader.

Navigating what’s to come

Few IT leaders anticipated a spike in IT salaries to this extent. Still, as the business and technology environment continues to remain unpredictable, CIOs must learn from what is transpiring with talent costs today to be better prepared to handle such scenarios in the future — especially as the CIO’s strategic becomes increasingly more important.

“They must use their expertise to anticipate future changes and work towards keeping the employees up to speed,” Alug advises. “Times are uncertain with the looming fears of recession and the ongoing geopolitical scenarios. In such situations, there’s no specific blueprint for CIOs to follow to predict upcoming changes. That said, their reliance on data and emerging technologies does take the driver’s seat to make calculative steps that can stave off any impending challenges.”

Salaries

The benefits of analyzing vast amounts of data, long-term or in real-time, has captured the attention of businesses of all sizes. Big data analytics has moved beyond the rarified domain of government and university research environments equipped with supercomputers to include businesses of all kinds that are using modern high performance computing (HPC) solutions to get their analytics jobs done. Its big data meets HPC ― otherwise known as high performance data analytics. 

Bigger, Faster, More Compute-intensive Data Analytics

Big data analytics has relied on HPC infrastructure for many years to handle data mining processes. Today, parallel processing solutions handle massive amounts of data and run powerful analytics software that uses artificial intelligence (AI) and machine learning (ML) for highly demanding jobs.

A report by Intersect360 Research found that “Traditionally, most HPC applications have been deterministic; given a set of inputs, the computer program performs calculations to determine an answer. Machine learning represents another type of applications that is experiential; the application makes predictions about new or current data based on patterns seen in the past.”

This shift to AI, ML, large data sets, and more compute-intensive analytical calculations has contributed to the growth of the global high performance data analytics market, which was valued at $48.28 billion in 2020 and is projected to grow to $187.57 billion in 2026, according to research by Mordor Intelligence. “Analytics and AI require immensely powerful processes across compute, networking and storage,” the report explained. “As a result, more companies are increasingly using HPC solutions for AI-enabled innovation and productivity.”

Benefits and ROI

Millions of businesses need to deploy advanced analytics at the speed of events. A subset of these organizations will require high performance data analytics solutions. Those HPC solutions and architectures will benefit from the integration of diverse datasets from on-premise to edge to cloud. The use of new sources of data from the Internet of Things to empower customer interactions and other departments will provide a further competitive advantage to many businesses. Simplified analytics platforms that are user-friendly resources open to every employee, customer, and partner will change the responsibilities and roles of countless professions.

How does a business calculate the return on investment (ROI) of high performance data analytics? It varies with different use cases.

For analytics used to help increase operational efficiency, key performance indicators (KPIs) contributing to ROI may include downtime, cost savings, time-to-market, and production volume. For sales and marketing, KPIs may include sales volume, average deal size, revenue by campaign, and churn rate. For analytics used to detect fraud, KPIs may include number of fraud attempts, chargebacks, and order approval rates. In a healthcare environment, analytics used to improve patient outcomes might include key performance indicators that track cost of care, emergency room wait times, hospital readmissions, and billing errors.

Customer Success Stories

Combining data analytics with HPC:

A technology firm applies AI, machine learning, and data analytics to client drug diversion data from acute, specialty, and long-term care facilities and delivers insights within five minutes of receiving new data while maintaining a HPC environment with 99.99% uptime to comply with service level agreements (SLAs).A research university was able to tap into 2 petabytes of data across two HPC clusters with 13,080 cores to create a mathematical model to predict behavior during the COVID-19 pandemic.A technology services provider is able to inspect 124 moving railcars ― a 120% reduction in inspection time ― and transmit results in eight minutes, based on processing and analyzing 1.31 terabytes of data per day.A race car designer is able to process and analyze 100,000 data points per second per car ― one billion in a two-hour race ― that are used by digital twins running hundreds of different race scenarios to inform design modifications and racing strategy.  Scientists at a university research center are able to utilize hundreds of terabytes of data, processed at I/O speeds of 200 Gbps, to conduct cosmological research into the origins of the universe.

Data Scientists are Part of the Equation

High performance data analytics is gaining stature as more and more data is being collected.  Beyond the data and HPC systems, it takes expertise to recognize and champion the value of this data. According to Datamation, “The rise of chief data officers and chief analytics officers is the clearest indication that analytics has moved from the backroom to the boardroom, and more and more often it’s data experts that are setting strategy.” 

No wonder skilled data analysts continue to be among the most in-demand professionals in the world. The U.S. Bureau of Labor Statistics predicts that the field will be among the fastest-growing occupations for the next decade, with 11.5 million new jobs by 2026. 

For more information read “Unleash data-driven insights and opportunities with analytics: How organizations are unlocking the value of their data capital from edge to core to cloud” from Dell Technologies. 

***

Intel® Technologies Move Analytics Forward

Data analytics is the key to unlocking the most value you can extract from data across your organization. To create a productive, cost-effective analytics strategy that gets results, you need high performance hardware that’s optimized to work with the software you use.

Modern data analytics spans a range of technologies, from dedicated analytics platforms and databases to deep learning and artificial intelligence (AI). Just starting out with analytics? Ready to evolve your analytics strategy or improve your data quality? There’s always room to grow, and Intel is ready to help. With a deep ecosystem of analytics technologies and partners, Intel accelerates the efforts of data scientists, analysts, and developers in every industry. Find out more about Intel advanced analytics.

Data Management

What is a data engineer?

Data engineers design, build, and optimize systems for data collection, storage, access, and analytics at scale. They create data pipelines used by data scientists, data-centric applications, and other data consumers.

This IT role requires a significant set of technical skills, including deep knowledge of SQL database design and multiple programming languages. Data engineers also need communication skills to work across departments and to understand what business leaders want to gain from the company’s large datasets.

Data engineers are often responsible for building algorithms for accessing raw data, but to do this, they need to understand a company’s or client’s objectives, as aligning data strategies with business goals is important, especially when large and complex datasets and databases are involved.

Data engineers must also know how to optimize data retrieval and how to develop dashboards, reports, and other visualizations for stakeholders. Depending on the organization, data engineers may also be responsible for communicating data trends. Larger organizations often have multiple data analysts or scientists to help understand data, whereas smaller companies might rely on a data engineer to work in both roles.

The data engineer role

According to Dataquest, there are three main roles that data engineers can fall into. These include:

Generalist: Data engineers who typically work for small teams or small companies wear many hats as one of the few “data-focused” people in the company. These generalists are often responsible for every step of the data process, from managing data to analyzing it. Dataquest says this is a good role for anyone looking to transition from data science to data engineering, as smaller businesses often don’t need to engineer for scale.Pipeline-centric: Often found in midsize companies, pipeline-centric data engineers work alongside data scientists to help make use of the data they collect. Pipeline-centric data engineers need “in-depth knowledge of distributed systems and computer science,” according to Dataquest.Database-centric: In larger organizations, where managing the flow of data is a full-time job, data engineers focus on analytics databases. Database-centric data engineers work with data warehouses across multiple databases and are responsible for developing table schemas.

Data engineer job description

Data engineers are responsible for managing and organizing data, while also keeping an eye out for trends or inconsistencies that will impact business goals. It’s a highly technical position, requiring experience and skills in areas such as programming, mathematics, and computer science. But data engineers also need soft skills to communicate data trends to others in the organization and to help the business make use of the data it collects. Some of the most common responsibilities for a data engineer include:

Develop, construct, test, and maintain architecturesAlign architecture with business requirementsData acquisitionDevelop data set processesUse programming language and toolsIdentify ways to improve data reliability, efficiency, and qualityConduct research for industry and business questionsUse large data sets to address business issuesDeploy sophisticated analytics programs, machine learning, and statistical methodsPrepare data for predictive and prescriptive modelingFind hidden patterns using dataUse data to discover tasks that can be automatedDeliver updates to stakeholders based on analytics

Data engineer vs. data scientist

Data engineers and data scientists often work closely together but serve very different functions. Data engineers are responsible for developing, testing, and maintaining data pipelines and data architectures. Data scientists use data science to discover insights from massive amounts of structured and unstructured data to shape or meet specific business needs and goals.

Data engineer vs. data architect

The data engineer and data architect roles are closely related and frequently confused. Data architects are senior visionaries who translate business requirements into technology requirements and define data standards and principles. They visualize and design an organization’s enterprise data management framework. Data engineers work with the data architect to create that vision, building and maintaining the data systems specified by the data architect’s data framework.

Data engineer salary

According to Glassdoor, the average salary for a data engineer is $117,671 per year, with a reported salary range of $87,000 to $174,000 depending on skills, experience, and location. Senior data engineers earn an average salary of $134,244 per year, while lead data engineers earn an average salary of $139,907 per year.

Here’s what some of the top tech companies pay their data engineers, on average, according to Glassdoor:

CompanyAverage annual salaryAmazon$130,787Apple$168,046Capital One$124,905Hewlett-Packard$94,142Meta$166,886IBM$100,936Target$183,819

Data engineer skills

The skills on your resume might impact your salary negotiations — in some cases by more than 15%. According to data from PayScale, the following data engineering skills are associated with a significant boost in reported salaries:

Ruby: +32%Oracle: +26%MapReduce: +26%JavaScript: +24%Amazon Redshift: +21%Apache Cassandra: +18%Apache Sqoop: +12%Data Quality: +11%Apache HBase: +10%Statistical Analysis: +10%

Data engineer certifications

Only a few certifications specific to data engineering are available, though there are plenty of data science and big data certifications to pick from if you want to expand beyond data engineering skills.

Still, to prove your merit as a data engineer, any one of these certifications will look great on your resume:

Amazon Web Services (AWS) Certified Data Analytics – SpecialtyCloudera Data Platform GeneralistData Science Council of America (DASCA) Associate Big Data EngineerGoogle Professional Data Engineer

For more on these and other related certifications, see “Top 8 data engineer and data architect certifications.”

Becoming a data engineer

Data engineers typically have a background in computer science, engineering, applied mathematics, or any other related IT field. Because the role requires heavy technical knowledge, aspiring data engineers might find that a bootcamp or certification alone won’t cut it against the competition. Most data engineering jobs require at least a relevant bachelor’s degree in a related discipline, according to PayScale.

You’ll need experience with multiple programming languages, including Python and Java, and knowledge of SQL database design. If you already have a background in IT or a related discipline such as mathematics or analytics, a bootcamp or certification can help tailor your resume to data engineering positions. For example, if you’ve worked in IT but haven’t held a specific data job, you could enroll in a data science bootcamp or get a data engineering certification to prove you have the skills on top of your other IT knowledge.

If you don’t have a background in tech or IT, you might need to enroll in an in-depth program to demonstrate your proficiency in the field or invest in an undergraduate program. If you have an undergraduate degree, but it’s not in a relevant field, you can always look into master’s programs in data analytics and data engineering.

Ultimately, it will depend on your situation and the types of jobs you have your eye on. Take time to browse job openings to see what companies are looking for, and that will give you a better idea of how your background can fit into that role.

Analytics, Careers, Data Management, Data Mining, Data Science, Staff Management