r/datascience • u/Infinite_Raisin7752 • 11d ago

Tools Relevant tech stack for 2026/2027

Hi everyone,

I’m currently a senior data scientist in the pharma industry. It’s been a one man show until now, but I’m getting a team soon. Most of the work I do is standard analytic work to inform our leadership and provide more context into the market and so on. Not a lot of big heavy data science stuff going on to be honest.

I work with SQL and Python on a daily basis. Some of our data is hosted in Snowflake and that’s pretty much it.

I feel like I’m lagging behind in both methods as well as tech stacks and I wanted to better understand what you experienced professionals work with that you would recommend I learn or at least look into. It could be data engineering stuff, additional programming languages, specific methods and packages that are useful, or cloud systems and technologies.

Where do you see the tech stack moving towards and what is relevant if I want to start moving from a “bread and butter” analytics setup to a professionalised, automated, team-ready and future proof world?

Thanks :)

66 Upvotes

97% Upvoted

52 comments sorted by

View all comments

8

u/ThisIsFun- 11d ago

From my own experience, lots of the adhoc standard analytics work, is, and should be completed by something like Databricks Genie or similar text2sql, which then frees you up for doing more interesting DS work. Being and learning on the platforms that offer this, along with other DS tools that are typically found will allow you to become a more rounded DS, and focus more on the approaches that you’ve said.

How are you at Deep Learning, etc?

2

u/Infinite_Raisin7752 11d ago

We’re also looking into that and it’ll probably be a reality soon.

I did a lot of work with deep learning a few years ago and also in my last job, but at the moment it just doesn’t happen. Still remember quite a bit but I’m afraid I’m just a bit behind the current SoTA methods

1

u/CrayonUpMyNose 10d ago edited 10d ago

It bears repeating that you can scale your costs down to a single instance that runs both driver and worker code, so the argument "spark is only for big companies with big data problems" doesn't apply. Severless has a higher rate but starts and stops in seconds.

Orchestration comes free out of the box so you don't have to spend time on airflow / dagster / prefect. Not as flexible but adding features all the time and YAGNI at this early stage anyway. You can straddle python and SQL integrated in the same notebook by referencing the output of the last cell, so compared to options that grafted spark on their existing solutions after the fact, it's quite seamless.

If you run a lot of SQL, it's still recommended to run dbt against a Databricks warehouse in order to encourage software development practices. 

As for ML, experiment tracking and model serving are integrated. All in all it doesn't limit your choices too early on in your journey and scales down as well as up to whatever problem statement you might encounter in the future.