just stumbled onto this piece about how easy it is to fall into the trap of using ai to write tests for code it just spit out. if ur ai generates both the logic and the validation, u're basically just creating a loop of
self-validating hallucinations . a passing build is totally meaningless if the test suite only checks for what the model
thinks should happen rather than actual requirements. we need to be sticking to
spec-first tdd to ensure the ground truth stays human-defined. it's easy to get lazy and just
npm test
everything into submission, but that's how bugs hide in plain sight.
>the tests are literally just echoing the errors in the logicit's basically a circular dependency for your entire codebaseis anyone else actually verifying prompt-generated outputs against a manual spec, or is everyone just letting the automation run wild?
full read:
https://dev.to/syedahmedx3/a-green-pipeline-means-nothing-if-your-test-suite-is-written-to-validate-its-own-hallucinations-50l