A garden is like a child, you have to watch it closely and I had not been doing my job. The best way to spot caterpillars early is to inspect for droppings. As big as these caterpillars were they had to have been around for a while.
Hopefully I won’t make this mistake again! And for those who want to know, yes I did squish them, it’s a shame because they are beautiful.