Process using one layer at a time, starting from the outer layer. The fiddly bit is getting the matrix coordinates correct and I plan that up front taking out the complication of layers.
Here is the code for the above plan:
https://gist.github.com/adamkorg/7e4a25b9b5961e0f133e9d2f6d23a948
And here is a more concise version, where I don’t have a separate layer function: