5 things that most data teams do NOT need in 2024:
1) Apache Iceberg
2) Real-time / streaming data
3) Self-service analytics
4) A data catalog
5) Data contracts
__
In the data world, right now is a pretty frustrating time for most data teams.
--> 10 person data teams have been cut to 3 people
--> 4 person data teams have been cut to 1 person
--> Many data teams have vanished entirely
And unfortunately, that means:
1. It's tougher to find jobs
2. The people that are still around are stuck doing more work
3. People are worried about job security
4. Small teams are managing over-complicated data infrastructure
5. Cost reduction is top of mind
6. Managing complexity is EVERYTHING
So, what can you do about it?
There are a handful of paths forward from here:
1. You roll up your sleeves and you dig yourself out of the sh*t that your team overbuilt during the last 5 years. This is a mentality shift more than anything...
2. You can quit (or quiet quit), and try and find a role in a company where they're starting from scratch on data infrastructure (sorry to burst your bubble, but most companies messed up bad over the last 5 years, so these situations are hard to come by)
3. Become a freelancer, and try and help companies from the outside (this is becoming very common right now)
4. Go to a bootcamp and become an AI person with dreams of making $1m / year training LLMs
5. Watch 'The Bear' and decide to become a chef
__
For most data people, these are all uncomfortable options.
If you've spent the last 5 years picking tools, hiring people, and ratcheting up your company's cloud bill, it's not fun to deprecate tools, fire people, or reduce cloud costs.
It's not fun to own up to mistakes that were made.
But that's the reality of where we are today.
2024 is not the time to pick fancy technologies or invest in infrastructure for infrastructure's sake.
It's time to get stuff done.
__
And if you haven't watched The Bear, I highly recommend it.
