Eric McDermid

At the core of effective engineering leadership is the ability to stay connected with your team. This is perhaps the most challenging aspect because retaining the respect of your people is difficult unless you remain technically involved yourself. In my journey as an engineering leader, staying true to the individual contributor role that originally earned their respect has kept me close. This approach extends beyond technical expertise; it's about maintaining a deep connection with the core activities that propel product development forward at Gearflow. One of my biggest professional fears is becoming irrelevant or disconnected from my team and our work. This fear motivates me to continually reinvent and retool in response to market demands. Next month, I'll be sharing about my adventures generating synthetic data recently. And while I'm no expert when I start preparing for these talks, the process is incredibly valuable. It's less about delivering a perfect talk and more about how the preparation expands my thinking and exposes me to new capabilities. This helps me connect the dots to opportunities at Gearflow that bring real innovation to our customers. Speaking regularly holds me accountable and pushes me to learn ahead of the curve, ensuring I remain relevant and impactful day-to-day. Every talk is a journey of discovery, helping me to lead with humility, communicate clearly, and inspire a culture of innovation through a deep, technical understanding of our work.

20

The final deep dive on change query support in the table formats is ready. This one covers Apache Paimon, focusing on understanding the change query support through the lens of its internals. https://lnkd.in/dFfM_8nA

28

Interesting post by João Salgado about partitioning vs. sharding in #Postgres which lists trade-offs for each. It's great looking at the list of cons for partitioning and seeing that #TimescaleDB actually helps with most of them - Complexity ⬇️ because we transparently manage partition creation and lifecycle for you as data is ingested - Optimization Challenges ⬇️ because we have planner hooks that help with handling data exclusion (you still do need to pay attention to your partitioning key though). - Storage Overhead ⬇️ (although Postgres doesn't really have much overhead for partitions) because you can just choose to compress your data (we often see a 95x reduction in size). Timescale 🚀

18

1 Comment

Explore the power of Thistle's CLI tools directly in your browser! Check out the latest blog post featuring our very own Russell Chapin at https://lnkd.in/g7fVUWN4. To experience the tools firsthand, visit https://demo.thistle.tech. #CLI #ThistleTech #BrowserTools

12

1 Comment

New blog post! ETL, ELT, and Reverse ETL are products of siloed teams and Conway's Law in action. To move forward, we need to replace the ELT mindset with the graph mindset—focusing on consumption and collaboration. https://lnkd.in/dqqR5tNM

72

4 Comments

I just published my latest blog post: "Experimenting with Rust, Nix, Parca & k6" https://lnkd.in/eUKeA57g This one is a long one, but it turned out well I think! #nix #rust #parca #blog

26

3 Comments

Overview of PostgreSQL in 2024. https://lnkd.in/dTCkYYTD

15

I've been testing GPT-4o since its recent release. While I haven't really had a use for the multi-modal (images, audio, and text) capabilities yet, the improvement in speed of response is noticeable. Not just noticeable from a spec sheet perspective either—it's noticeable in how the product works in real use cases. I've been thinking about why that is, and here's my theory: there's a breakpoint somewhere based on latency and response speed. Above this breakpoint, interactions feel sluggish and disruptive. Below it, they become seamless and effortless, akin to a natural conversation. Previously, I posted about how Llama and Gemini models seemed noticeably more usable than GPT-4, with my preference being Llama whenever possible. This new experience with GPT-4o further emphasizes the importance of optimizing both latency and throughput in AI systems. This chart from ArtificialAnalysis.ai shows that GPT-4o has a total response time of 1.5 seconds for 100 tokens, compared to GPT-4's 5.7 seconds. This speed difference isn't just a matter of numbers—it's fundamental to achieving a smooth and efficient user experience. When latency drops below a certain threshold, the interaction becomes fluid and engaging, making the AI more practical and user-friendly. It also makes its mistakes more tolerable. This improvement transforms how we use these models in real-time applications, making AI interactions faster and more intuitive. It underscores that achieving low latency and high throughput is no longer a nice-to-have but a critical requirement for practical AI deployment.

9

3 Comments

These are the boring, yet impactful aspects of API operations I am here for--low-tech, economy impacting, real-world stuff like the API deprecation notice from the Department of Labor.

20

I had the same thought. Polaris is another data catalog that lives in your Snowflake account and the sync with Snowflake data catalog is manual. I would love it if I could access all Snowflake tables through Polaris. That being said, the main use-case Polaris unblocks is to be able to run Spark on top of Snowflake (Iceberg) tables hassle-free and efficiently, there is no need to set up S3 access outside of Snowflake. It's a big deal because you don't need to migrate from Spark to Snowpark unless it's much faster and can use Trino / DuckDB on top of your Snowflake tables. Given that 40% of Snowflake customers are also Databricks customers, that's a good gesture as Snowflake seems to believe in their performance & cost efficiency on the compute layer.

7

*** An IMPORTANT Public Service Announcement *** There exists a copyright trolling company, PicRights, whose entire business model is legalized extortion. Be VERY CAREFUL that you do not get targeted by them! And be VERY CAREFUL to not unwittingly hire someone who has ever worked for PicRights! PicRights nominally targets image copyright infringers on behalf of its clients. In reality, PicRights targets and extorts innocent people and organizations by asserting copyright infringement even in cases of legally protected non-commercial fair use. If you have ever shared a meme on social media or in an online discussion forum, or if you have ever used images from Google search results to build a little web site or blog for yourself or for any kind of non-commercial organization (whether registered as a non-profit or not), you are likely to be targeted and extorted by PicRights. PicRights threatens innocent and unsuspecting individuals and small businesses with prohibitively expensive copyright lawsuits if they do not agree to pay a settlement amount. The amount varies from case to case, but it is intentionally set at just less than the going market rates of qualified attorneys who could mount a valid legal defense. PicRights does this knowing that it forces their victims into a financial and legal corner that can only be escaped by paying the settlement fee. Simply removing the copyright-claimed images is not enough -- they demand an absurdly unreasonable payment to settle the matter if you want to avoid having your entire life ruined by a hopelessly expensive lawsuit. Even if your usage is legally protected fair use, it won't matter, because it will cost you far more to defend yourself in court than to pay the extortative settlement fee. Everyone -- from high school students using social media and hobbyists running personal blogs, to entrepeneurs and CEOs of respectable companies -- needs to make themselves aware of this company and its extortion scheme. Once you are targeted, there is no financially viable way to get your life back other than to pay PicRights' ransom demand. If you run a respectable law firm or company that deals with intellectual property law, you should make yourself aware of what PicRights does and how it ruins people's lives and financially destroys organizations and companies, and you should be very careful to avoid hiring ANYONE who has EVER worked for that company in ANY capacity. (FYI: One such PicRights employee is this gentleman: https://lnkd.in/g2cc3jiS ) If you run a respectable company that sells commercial usage licenses for your photos or other digital media, you should make yourself fully aware of the morally corrupt extortion tactics utilized by PicRights (or other similar companies) before hiring them to represent you. If you're a law firm looking for a situation ripe for a class action lawsuit, look into the business practices of PicRights.

8

I share because the meta-message of making sure you've done the appropriate level of diligence before picking your database (or any architecturally significant component) is a much-needed message IMHO. I've seen too many examples in the past few years of decisions made without the appropriate level of diligence and any regard for the unnecessary costs these decisions impose on the business. https://lnkd.in/eD_24953

4

Confluent's Tableflow is getting closer and closer to GA. Materializing Kafka topics as Iceberg tables, but with a great deal of thought around schema conversion, schema evolution and such. It's amazing how much work goes into making a product robust to schema changes. After the initial announcement I wrote two articles explaining Tableflow and its role in the future of bringing operational and analytics planes together. * Tableflow: the stream/table, Kafka/Iceberg duality: https://lnkd.in/gZbQ2cZ8 * Hybrid Transactional/Analytical Storage: https://lnkd.in/dTg6DKVy

102

4 Comments

Did you miss my final #RAGHack stream, "Evaluating your RAG chat app"? Catch the recording here: 👇 https://lnkd.in/gSd32wXm I covered: 🤔 What does it mean for a RAG answer to be high quality? 🔍 Which factors affect RAG answer quality, and which affect it the most? 💻 Using promptflow-evals SDK to run GPT evaluators in Python 📊 Using my ai-rag-chat-evaluator CLI tools to generate synthetic ground truth data, run batch evaluations, and compare metrics across runs 🤷🏻‍♀️ Strategies to make sure your app can say "I don't know" at the appropriate times Grab slides from https://lnkd.in/g-9xSQvV #rag #openai #python #evals

20

My takeaway from Tidelift’s latest annual "state of the open source software maintainer" report? We’ve got a long way to go as an industry to better partner with maintainers, but we’re making progress. Highlights: 🔒 Maintainers are spending 3x more time on security than 3 years ago. ✅ Paid maintainers spend more time on security, docs & maintenance than unpaid volunteers. 💰 Who’s paying the maintainers? Sponsorships, employers, and Tidelift. 👻 The XZ utils incident is making maintainers more guarded. 🤖 Maintainers are skeptical to cautiously optimistic about AI, for specific good reasons Read the full report based on responses from over 400 maintainers. https://lnkd.in/eHhiF9dJ

11

3 Comments

OK, I'm a little late to the party, but still just as excited to share that we published our third Tidelift state of the open source maintainer report last week! I am super grateful to have been part of the team that worked on this, and to have had the opportunity to learn from Chris Grams and Lawrence Hecht about gathering and analyzing thousands of data points that went into this report. Leveling the playing field between open source creators and maintainers and the large enterprise organizations that use their software is a mission dear to my heart. This report helps us at Tidelift and the industry at large better understand the gaps and to work towards addressing those gaps. Some of the key findings include: - 60% of maintainers are (still) not paid for their work - Maintainers are spending 3× more time on security than they did a few years ago - Paid maintainers are significantly more likely to implement critical security practices than unpaid maintainers - Almost half of maintainers feel under-appreciated and like the work is thankless - And lots more! Download the full report here -> https://lnkd.in/eDWSxbMy #openSource #IndustryInsights #opensourcemaintainers

15

My Apache Hudi CDC first draft is sitting in my blog ready to publish. I'll reread it with fresh eyes on Monday and publish.

18