just stumbled onto a decent workflow for training models when base versions like
llama 3 or
mistral aren't cutting it for niche tasks. if you're dealing with stuff like medical coding or financial summarization, generic weights usually miss the mark. using
databricks mlflow and
spark seems to be the move for handling the heavy lifting when your dataset is too massive for a single node.
it basically lets you adapt those pre-trained weights using your own proprietary labels at scale. has anyone here actually tried moving this pipeline into production, or are you still sticking to simple prompting?
too much infra overhead is the main concern i have with this approach.
article:
https://dzone.com/articles/llm-finetuning-databricks