Exploring the power of contextual meaning in data chunking has been truly transformative. By leveraging techniques like calculating Euclidean distance between embeddings, we're able to chunk content based on its inherent meaning, removing the burden of arbitrary chunk sizes. It's fascinating to see how traditional machine learning libraries like scikit-learn remain invaluable even as we delve deeper into Generative AI.
Recently, I revisited scikit-learn to calculate cosine similarity between pairs of sentences, reaffirming the importance of mastering foundational machine learning and deep learning concepts.
from sklearn.metrics.pairwise import cosine_similarity
As we push the boundaries of Generative AI, it's clear that our journey is rooted in traditional methodologies. Understanding and mastering these foundations is essential for unlocking the full potential of Generative AI.
#genai #datascience
Esteemed telecom Evangelist | Expertise in Private & Public Cloud Architectures | Seasoned Prompt engineer utilising Generative AI/RAG & Vector Embedding. Leveraging business process automation with frequent improvements
2moCongrats Malachy Lavelle this course from LinkedIn seems very insightful. Would you like to talk live on this with me? 🙏, lets #collaboratetoelaborate on a talk show episode with me and also please connect on my 2nd account as this account i have already reach 30 K connection which is the max limit of linked in connection so i have been creating other accounts to accommodate more 30L connection per account limit. see you on Dr. Ankur Rajan Verma