You can’t extract information about the inner workings of a human without interacting with live humans or live parts of humans. And pretty much everything that happens inside humans is very very context dependent so you need in vivo information to get anywhere.
Yes, this applies to humans just as it does for automated systems. I don't see a specific argument here.
but you’d still need access to humans to do the testing or to confirm the results of simulations.
I can't see it. The testing can be automated. There is no need to restrict automation to simulations.
Probably I'm too naive, but I think there are some rather different areas here that need to be discussed.
One of applied physics where you have sufficiently accurate models that are sufficiently tractable and also sufficiently stable to changes in input, which it seems can be dealt with at increasing ease. That is where progress has been in recent months. The descriptive nature of the major history of biological research will probably bide some time for biologists, but I don't see why that would be too crucial in the greater scheme of things.
There will also always remain areas, for example a naive guess would be many areas relating to quantum field theory, where large parts of rigorous tractability still breaks down for rather theoretical reasons, and there is a need for "more". That however is also the case without AI.
The other area lies in robotics i believe, where due to inaccurate models or ones where accuracy of the model is rather irrelevant due to the complexity and instability of the dynamics of the models, other processes have to drive advancement. Advancements in robotics may be harder to predict but much of it seems to rely heavily on programming which is precisely where progress is happening. I am still to come across any theoretical arguments on limitations that make sense and previous hypothetical arguments seemed to have all rather fallen apart quickly. I don't think current benchmarks are anymore a meaningful argument.
I'm yet to see a deeper discussion amongst biologists just as in other disciplines that goes beyond the, in my eyes rather meaningless, "we are dealing with many millions of complex interactions" and where the other side isn't just Silicon Valley goggleboggle talk that claims that shortly there will be an "AI overlord" that solves ME/CFS. I think this is a rather large problem as these processes are seemingly beginning to shape our society and we want that disruptive impact to be positive.