It’s mostly about the assumptions that have to be fulfilled for NNs to be able to solve a problem: that there is a connection between the data and whatever you want the model to explain with the given data.
If there is no connection, it can’t be explained with that data. At least that’s my basic understanding of it.
The UAT is not a result of the form: This and this has to apply for NNs to "work". It is rather exactly the opposite. Instead of giving you conditions on when things work it simply tells you that there always exists at least one thing that does work but that thing might depend on the nature of the object your looking for. It tells you: No matter how your data looks like you can find a NN to approximate the function underlying said data. The only assumption is that the function is continuous and conditions on what type of NN you allow but those things can all be adapted. Naturally for a given NN you can always construct a function that cannot be approximated well, but that is a different statement. My understaning is that the UAT is not very surprising, rather it is to be expected. The perhaps surprising thing is rather that in many cases (for example inverse problems in imaging) things often work quite well even once the NN has been "fixed" (this however requires a rather different architecture than the one used in the original UAT).
That is, it is an existence result. That roughly means given an arbitrary function there exists a NN with one layer to approximate said function, so in short it says "no matter how complex the pattern you are looking for is there exists a NN that will get you there for this one specific pattern", so a different pattern might require a different NN and so forth (but there are some bounds and convergence results depending on specific setups). But simply because it tells you there exists one thing that gets the job done does not rule out the fact there don't exist other things that get the job done much better.
A priori that tells you nothing about how fluctuations in the data effect choice of network or how well your given network still performs, it tells you nothing about there not existing a different network architecture that gets the job done better or there not existing millions of networks that all do the job or that there aren't better architectures which outperform the pre-specified architecture in the UAT.
All of this is pretty much orthogonal to what
@mariovitali is doing because he is given a fixed NN, whose architecture is not even publicly known, and trying to approximate various different functions as good as possible, knowing that the function he wants to approximate in most cases does in fact not match the data precisely because that data is noisy. So you just end up approximating functions you don't want to approximate even though there exists something that would have approximated precisely the things you wanted to approximate! In this case you can be a priori certain that you can construct a function that this NN cannot approximate but you also have no idea what the function you're actually looking to approximate looks like.
In short: Noisy data is a whole different can of worms that the UAT does not deal with because with the UAT you are assuming that the function you want to approximate is precisely the function you want to approximate, rather than asking questions about stability (and we already know that the NNs of the original UAT are not stable in a certain sense).
I’m pretty sure new strategies were developed as a result of the introduction of AI into chess bots, that had not been found with previous methods. Some even goes completely against all conventional wisdom, but they still work somehow.
Yes of course, nobody is denying that. I don't understand much of chess nor what happened but it seemed that agressive positional advantages and lines were rather underestimated by humans as well as brute force engines such as Stockfish which were seemingly trained on the value of pieces.
I’m not sure I understand what jump means here.
You are not alone with that. Neither do I. Perhaps sometime I will, but rather likely I will never be able to make the "jump" to get there. But it is clear that it is a form of abstraction that is different to the one of playing a better game of chess once you know the rules and perhaps more similar to coming up with the rules of chess after only having watched one game of chess, but still a bit more involed and different to that. So in some sense a deeper understanding of something that does not simply follow from a straightforward albeit lengthy computation. Perhaps it is more comparable to coming up with and proving the UAT because you see a use for it, after only having been told how
Giuseppe Peano counts his sheep at night, which of course von Neumann would be able to do, despite him possibly having only understood what an abstraction looks like which made it sufficient for him to be able to make completely different abstractions elsewhere.
