finally moving away from those shitty prompt hacks for json outputs. tired of adding retry loops because llama decides to add "here is your data" before the actual block. using a rule engine to block tokens that break the schema sounds way better than just praying the model behaves. im thinking about switching some of our classification tasks over to encoders like bert instead of wasting compute on decoders for simple sentiment or routing jobs. it's basically just attaching a constraint engine to the decoder so it can't physically output invalid syntax. might save us some headache with those broken pipelines. anyway, seeing if this actually scales without killing latency.
more here:
https://dzone.com/articles/constrained-decoding-models-explained