Assigned
Status Update
No update yet.
Comments
va...@google.com
si...@google.com
This feature request has been forwarded to the Data Fusion engineering team so that they may evaluate it. Note that there are no ETAs or guarantees of implementation for feature requests. All communication regarding this feature request is to be done here.
ga...@gmail.com
Struct support is being prioritized and can be tracked in open source by https://issues.cask.co/browse/CDAP-15349 .
si...@google.com
Hello,
Thank you for your response! This has been forwarded to the Cloud DataFlow Engineering Team so that they may evaluate it. Note that there are no ETAs or guarantees of implementation for feature requests. All communication regarding this feature request is to be done here.
Description
What you would like to accomplish:
- Leverage LLMs capabilities for data processing and create RAG based pipelines in dataflow and apache beam.
- Create vector embeddings for text directly within the pipeline.
How this might work:
langchain-beam java library is a apache beam and langchain integration. it provides transforms to integrate LLMs and embeddings models from gemini, openai and other providers in beam pipeline.
Goal is expose those expose those transform as template for easy integration in Dataflow.
repository link -