Benchmarking Chain-of-Thought Prompting

Chain-of-thought (CoT) prompting has become a go-to technique for improving LLM reasoning. But how well does it really work?

The Study

I tested CoT prompting on a range of models and tasks, from math to commonsense reasoning.

Q: If there are 3 red balls and 2 blue balls in a bag, what is the probability of picking a red ball?
A: Let's think step by step. There are 5 balls in total...

Findings

• Larger models benefit more from CoT
• Some tasks (like math) see bigger gains
• Prompt phrasing matters a lot

"The biggest surprise: small models sometimes get worse with CoT!"

Conclusion

CoT is a powerful tool, but it's not a silver bullet. Use it thoughtfully and test on your own data.