There is a mantra in the programming world that states “You can’t retrofit performance”.
The standard approach is to write all project code, to get a logically correct product, and, once complete, then try to improve the performance.

This would be like designing a family saloon car, and then trying to incrementally improve each component to (eventually) produce a Formula 1 (F1) race car.

At this late stage, every component inherently has acquired the low specification of the initial product. Each component depends to some extent on every other component its connected to. This is the wrong approach.

If you want to compete at F1 level, you must start with every component being of F1 calibre.
If you are looking for better than F1 performance, you need an entirely new approach.