Many of my mentees ask me whether a career in Data Engineering (DE) or Data Science (DS) is better. This is a badly posed question. These roles are sometimes seen as the two extremes within the Analytics field. However, in my opinion, it mostly depends on where you are working and the maturity of your company.
“Data is the new oil,” and both these roles, with their many facets, are important to efficiently manage and strategically use the vast amount of data that is available nowadays.
In this blog post, we’ll explore the differences between DE and DS and which career path might be right for you.
What is Data Engineering?
Data Engineering is a field that focuses on the design, development, and maintenance of data systems. Data engineers are responsible for building and managing the infrastructure that allows organizations to collect, store, and process data. They work with data architects and data scientists to ensure that data is properly ingested, transformed, and stored.
To be successful in DE, you need to be passionate about database design, data modeling, data warehousing, data integration, and ETL (extract, transform, load) processes (although recently the ETL paradigm has shifted more and more towards the ELT paradigm, more on this in a later post). Data Engineers must be proficient (or be willing to adapt) with a variety of tools, including SQL, NoSQL databases, Hadoop, Spark, and cloud-based platforms like AWS, Google Cloud, and Azure.
The technology stack depends on the company and its maturity, and it’s always changing. In my experience, a DE task can be performed directly within the database, using cloud services (e.g., AWS, Azure), Python scripting, or a different combination of all of the above.
What is Data Science?
Data Scientists are the superheroes of business insights. They tend to focus more on theoretical foundations such as statistical analysis, machine learning, and other techniques to analyze data and uncover patterns and trends that can inform business decisions. A critical, although oftentimes underrated, skill for DS is stakeholder management. A successful DS role must be able to relate to and understand the business profoundly, to effectively leverage the data in ways that make sense for the business. Communication, storytelling, and not being an ass is paramount!
The toolkit of DS, as mentioned, is mostly based on theoretical grounds, thus the technical tool used (being Python, R, SAS, or whatever) is less important (especially at the beginning of your career, when I think you should be focusing on learning as many tools as possible). There are some requirements to access this space, where you are expected to be knowledgeable in statistics, machine learning, programming, data visualization, and storytelling.
From my personal experience, the role of a data scientist is often misunderstood. Often times, the job description requires statistics, experience with machine learning libraries, and Python programming. Most likely, you will be using a lot of SQL to clean the data. I believe it’s rare that a data scientist has the ability and time to dedicate to pure research-driven projects, and often, you will need to bring value to the company by leveraging quick and dirty analysis, producing reports, and making sense of chaotic data.
Key Differences between DE and DS:
The key difference between DE and DS is their focus. DE is all about building and maintaining the infrastructure that allows organizations to collect, store, and process data, while DS is all about analyzing data and extracting insights.
Another key difference is their workflow. DE is a more sequential process, with clear steps to follow to ensure the accuracy and reliability of the data. In contrast, DS is an iterative process that involves exploring and testing different models and algorithms to find the best fit.
In my personal experience, a DS must be okay with uncertainty, with a high degree of freedom that is counterbalanced by the unknown factor of bringing value to the business with a specific analysis and with the constant back-and-forth between business needs (which are pragmatic, timed, and very much goal-oriented) and the aspirational ideal case scenario where the DS is given the freedom to explore and put into practice pure Research and Development cycles.
The Data Engineer, on the other hand, doesn’t often go towards a quest for finding the golden pot at the end of the rainbow. Their goals and accomplishments are likely well described and set given the environment and the business needs (given that, based on the company’s maturity, this may not always be true). The path for a data engineer is often clearer because their requirements are more defined.
The ambiguity of the DS role comes with some benefits, though. DSs are often highly regarded (especially if successful!) by business stakeholders, and they might get a lot more spotlight, which in turn provides nice opportunities for professional growth (aka promotions!). DEs are, as I said in a previous post, unsung heroes. They build and maintain infrastructure and, despite the fact that without them, advanced analytics could be difficult or even unachievable, they don’t have a lot of interactions with business stakeholders. This often has repercussions on their progression velocity and career opportunities.
Which Career Path is Right for You?
Deciding between DE and DS can be challenging, especially if you’re new to the data field. To make an informed decision, you should consider your strengths, interests, and goals.
If you enjoy working with databases, data storage, and data processing, and have a passion for building scalable and efficient data systems, DE might be the right career path for you.
If you enjoy statistical analysis, machine learning, and data visualization, and have a passion for uncovering insights and telling stories with data, DS might be the right career path for you.
Is a Career in DE or DS Worth More?
The answer to this type of question is never easy and possibly not even helpful. As mentioned above, you should probably consider one career versus the other based on your attitudes and what makes you happy and passionate.
Below is my personal take on the current status of these roles.
What I’m observing is that the Data Scientist role is becoming less and less relevant. A couple of years ago, the Data Scientist role was all the rage in the tech industry (which is often spearheading new trends). Data scientist positions were blooming, with a pay range that was incredibly high.
Nowadays, my intuition is that the DS role is more and more being downgraded to a Data Analyst role. This is probably due to multiple reasons:
- The DS role is incredibly expensive but often doesn’t carry a net value for the company. The pure DS role is very research-driven, so the cost/benefit of this role might not be apparent right away, especially if the company is not mature enough in their data infrastructure.
- More and more low-code and ML services are coming into play that allow seamless (or almost seamless) integration of the data, providing a stable and fairly accurate output, without the need for in-house data scientists to produce custom-made, ad-hoc modeling.
- Often, at least in my experience, companies might not have the right data infrastructure yet. This entails a lot of data quality, data processing, data reporting before even starting with advanced analytics. These skills and experiences are already possessed by Data Analysts. Companies can thus hire Data Analyst talent, paying a much less expensive bill.
On the other hand, I perceive an increase in specialization on the DE side. This is not a bad thing and, in my opinion, is going to drive even more opportunities for this field. Roles such as Analytics Engineers, DataOps, and specialist Data Engineers will become more and more important, giving more opportunities both to people trying to enter this field as well as people with experience to further specialize or upskill.
My final thought is that you should invest in what makes you happy and, ultimately, what drives you out of bed in the morning.
Also, don’t fall for the fragile loop of always seeking the latest buzzword in the job market. It might go well and get you a pay raise once, but in the long run, I truly believe that if you are passionate, hard-working, and constantly curious, you will be able to stay relevant and adapt accordingly to the job market’s requirements.
Still not sure? I’m always happy to dive deeper into your specific use case. Reach out and we’ll schedule an initial call to understand where you are, where you want to go and how I can help in reaching your goals!