How to loose 36 million in a year
Release radar
New models
We are back in the stairs of Hyllie Library in Malmö and in the 60 days since last time, more than 25 frontier labs have released new AI models. So, of course the guys at Nordaxon wanted to do a visual comparison between the most prominent ones. They settled for using Simon Willson's plikan test. Though instead of a pelican on a bike the prompt said:
Draw a monkey sitting on top of a wooden barrel.
The monkey should look relaxed and content.
The words "Barrel AI" should appear clearly on the barrel.
Of course we would love to be able to brag with some great European models. But when it comes to monkeys, the best we got seems a bit behind still.
Claud Mythos
Obviously Mythos was also mentioned and even though there is still a lot of secrecy around the findings made by that model, the number of security bug fixes per month fixed in Firefox, is probably a hint of some sort. Though in discussions later some of us monkeys speculated that it might also be a mixed bag of issues that previously had lowe priority and now we all need to up our game on security, but in a few month the status quo might have caught up and most of the systems might again have less issues and be a lot safer, just by necessity. It is like someone just raised the bar on want need to be fixed now and what can stay in the backlog a few more months.
Andon Labs
The project that starten in San Francisco got a cafe in Stockholm and as with most of the attempts of letting AI run businesses without supervision, a bunch of... interesting decisions were made. Read more about it at andonlabs.com/blog/ai-cafe-stockholm.
If you want a more regular dose of release radar we encourage you to head over to radar.nordaxon.com where you will find weekly updates with AI news.
How to lose 36 million in 1 year
So, to the presentation that contrary to most financial advisors promised to teach us how to lose millions. Though this was more like an exhibition of how physics can be used with probabilities to get Neural Nets to qualify their responses, implemented to predict long term trends in the stock market. Michael Green was clear: You cannot predict the stock market. But you can, with the right information and a focus on how uncertain the predictions are, you can tease out trends in the data that you then can use as a basis for decision on how much to bet and for how long. And in that way AI Alpha Lab have gained a slight advantage over their competitors. But Green empecises that it is probably just a matter of time before more researchers start modeling with uncertainty and then will be able to reach similar results.
The major problem that Michel is focused on in most if not all of his projects is to get a probability distribution of likelihoods rather than a single number from a model that behaves like it is confident even though it really has no clue. There are two important flavours of uncertainty that you should be aware of. The first is Aleatoric, the noise in the data itself. More data like this will not improve your result, but it is often hard to reduce. The second flavour is Epistemic, things the model does not know. You want to minimize this with better data and here is where things can get expensive. There are always things that you nor the model knows. So, how do you minimize the unknown?
After a few more technical slides and thorough explanations this was the conclusion that depending on your need, this is the stack that Michael recommends:
- A heteroscedastic head for the aleatoric part. Essentially free.
- A deep ensemble with N ≈ 5 to 10 for epistemic uncertainty.
- Laplace when that ensamble is too expensive.
- A conformal wrapper on to, to check calibration.
After the presentation the audience had many questions on this physics heavy topic as well on the quirks of the stock market.
You really missed out if you did not come. So, make sure to keep an eye out for our next event and subscribe if you haven't already.
-
Johan Broddfelt
Probebalistic, bayesian, LLM