The idea of the ABS being the culprit makes some sense, but then again, lots of Mustangs out racing with slicks and big brake kits with aggressive pads, that do not have this problem. I think Blacksheeps advice to look at something in the rear of the car sounds to me to be the most likely cause..Not sure I can point you in the right direction, only tell you how I solved what sounds like a similar problem. As I came up through the HDPE program, my braking was always smooth and firm. Of course no where near threshold, but I had lots of confidence in my braking. But as part of my effort to get better, I went from Hawk HP+ front and rear to a very aggressive Hawk HT-10 front and rear. That is where by problem started.
The car started to feel loose under braking, and one day while trying to chase down my buddy, I did a big swap under braking coming off the back straight at UMC to a hairpin. It made for some dramatic video. Luckily I went two feet in and the corner had plenty of run out. Lost the corner, saved the car kind of thing.
That led to about two years of trying different pads, different techniques to try to get my braking mojo back. Lap times stagnated. So I decided to go back to the basics, go back to a pad set that I knew would only be slightly more aggressive than my HP+ days. For the last year I have been running Hawk Blue fronts and HP+ rears. I know not a popular combination here at TMO.
Everyone on this forum will likely think I am crazy, but the step back has worked. It allowed me to work on my technique with a familiar feel to braking. The car feels more planted and as I have stuck with it, my braking has improved greatly. Lap times have dropped. Five personal bests in the last 5 sessions works for me.
I have a personal theory, that others can feel free to chime in on. I think that the ABS logic in my car could not keep up with the very aggressive pads, having been engineered for street use. Going back to a much less aggressive pad put me back in the happy place for my ABS and allowed me to work on the loose nut behind the wheel. Again, just a theory.
I hope you get it figured out because I know what you are going through!