Introducing ITC Studio: A Tool To Scale Community Innovation On AI Strategies
Opening up the third scaling law to everyone
Today we are excited to launch the alpha version of ITC Studio, and a Reddit community called inferencesystems to discuss this type of work. This tool makes it very easy to test out various inference time compute strategies on different LLMs. There are 3 reasons we believe this is an important product to put out into the AI ecosystem.
Inference Time Compute is still a largely unexplored space. Providing an easy way to experiment will help move the ecosystem forward.
Few people have developed a solid intuition yet about what ITC strategies map best to certain models, benchmarks, and desired outcomes. Experimenting with our ITC studio can help companies and developers figure that out more easily.
AI innovation will move faster if there is more experimentation. Experimenting on the model or pre-training side is difficult because it requires so much data and compute. It remains expensive and out of the reach of most organizations. Experimenting with post-model improvement tactics is much more manageable.
I want to expand on point 3 a bit and reference a very popular article written last week by the former VC Jerry Neumann entitled “AI Will Not Make You Rich.” In the piece Neumann writes the following (bolded sentence is my highlighting):
Let’s grant that generative AI is revolutionary (but also that, as is becoming increasingly clear, this particular tech is now already in an evolutionary stage). It will create a lot of value for the economy, and investors hope to capture some of it. When, who, and how depends on whether AI is the end of the ICT wave, or the beginning of a new one.
If AI had started a new wave, there would have been an extended period of uncertainty and experimentation. There would have been a population of early adopters experimenting with their own models. When thousands or millions of tinkerers use the tech to solve problems in entirely new ways, its uses proliferate. But because they are using models owned by the big AI companies, their ability to fully experiment is limited to what’s allowed by the incumbents, who have no desire to permit an extended challenge to the status quo.
This doesn’t mean AI can’t start the next technological revolution. It might, if experimentation becomes cheap, distributed and permissionless—like Wozniak cobbling together computers in his garage, Ford building his first internal combustion engine in his kitchen, or Trevithick building his high-pressure steam engine as soon as James Watt’s patents expired. When any would-be innovator can build and train an LLM on their laptop and put it to use in any way their imagination dictates, it might be the seed of the next big set of changes—something revolutionary rather than evolutionary. But until and unless that happens, there can be no irruption.
We agree. As it stands, AI experimentation - in terms of actually making things smarter - is a small and controlled industry. Our hope and belief is that if inference time compute really is the third scaling law, then making it easier to explore - for everyone, will kick off a new wave of AI experimentation.
Over the coming months you will see this tool evolve in two directions. The first is more components of ITC strategies you can test. So expect Beam Search, Best of N, Tree of Thought, Chain of Thought, and many other tactics to be added soon. We are working on them. If you have something you really want us to add - email us and let us know, or leave a comment here.
The second direction is that we will make available (considering open source, but will see where we land) the back end tool that can automate running of many iterations of strategies. So if you are one of those rare companies that is advanced enough in your AI journey to be considering a multi-model systems level implementation, and have eval sets to easily evaluate the results of these tests, this tool will allow you to quickly test thousands of configurations. If you fit that bill reach out and we can work with you to try it out.
The AI industry is at a critical point and we have felt for months that not enough focus has been paid to inference time compute algorithms and the impact they can have on cost, performance, accuracy, and many other system level AI metrics.
We also believe that over time, as the model market commoditizes and fragments, that intelligence will move to the systems level and that a “meta intelligence” approach of understanding what models and tools the system has available will be necessary to best optimize any AI implementation. Neurometric is one of the companies building to that future.
Please reach out if we can be helpful with your AI problems. You can try the new tool here.