Rendered at 09:39:05 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
iamcoder18 9 hours ago [-]
> “I think I might need some relaxation time. It feels important to take a breather and find ways to unwind. There’s a lot going on sometimes, and it’s easy to forget to slow down. Maybe I could explore some activities that help clear my mind or consider options like a calming walk, some quiet reading, or just reflecting on things that bring me joy. It’s all about finding that balance, right?”
— GPT-5.6 Terra, reasoning summary, mid-run
This is hilarious
foven 22 hours ago [-]
I've seen this concept of using LLM/AI/etc for high throughput discovery of materials so, so often in the past 5 or so years and yet there hasn't really been any impact as a result.
I think this is the first one that has actually taken the pain to say how many of the discovered materials are actually feasible which is a real step in the right direction. Probably worth keeping in mind the step beyond plausible synthesis which is the actual cost/effort of the material. There's not much point if you find out RuO2 would be better than SiO2, as an example, if Ru is orders of magnitude more expensive.
A challenge I think you'll run into is that I expect the biggest companies (e.g. IBM) will already be doing the part they need themselves. I heard tell of IBM in particular using ML to improve their own chips before LLMs came along, so I'd be shocked if these bigger companies weren't already doing this for their own problems. Also, if you aren't doing the experiments yourself, it's always going to be a challenge to find a partner to test things for you and this will probably be the major time sink.
advaith08 20 hours ago [-]
good points. one of the reasons we picked the semiconductor industry is that its less price sensitive than others-companies are willing to pay if the performance is there. Effort is a different story though, and definitely a tradeoff to keep in mind.
We're doing experiments ourselves now at university partner labs (UC Berkeley and Stanford), which helps us get moving quickly. At some point, we'll need a partner though - the equipment and testing process quickly get very expensive.
brennanpeterson 10 hours ago [-]
Semiconductors are also incredibly risk averse, and requires fairly large gains to be worth the risk.
There are domains (or, mems, photonic sensing, high power) where innovation has a bit more flexibility for the early going.
advaith08 6 hours ago [-]
that's true, we've seen examples of many promising startups that are a few years in and stuck because the industry is so risk averse. will explore the other domains you've mentioned!
timr 10 hours ago [-]
Interesting. Far from a domain expert in materials science, but I've worked professionally in this area. What's your method for identifying valid "novel" compounds? Certainly, anything actually novel has been included in the models' training set already, unless you're doing a CASP-like coordinated blind test...right?
As an aside, the "Fable lies and cheats" section made me laugh -- have encountered this same failure mode, albeit for much simpler models. Polymerization (I realize this is not exactly the same thing) is an unbounded 3D playing field for constraint escape. You can try to put an additional constraint on polymerization, but then it will just make minor variations on the monomers...
akashramdas 7 hours ago [-]
So novel is defined as just anything that has not been synthesized as a thin film before. These are usually structures (phases or polymorphs) of materials that have not been reported in the literature.
We did find there are ways we can prevent this particular reward hacking, just consistently surprised by how these models can find tricks to technically achieve the set goal
praccu 6 hours ago [-]
Cool stuff. I first worked on ML for exploratory synthesis in 2012, and am still in related areas.
Once you have the experimental loop running, I suspect it will be quite difficult to hill climb on this task.
There will be some improvements you can make to the harness, but I suspect you'll be doing a lot of human in the loop review and providing feedback that goes back into the harness instructions.
I know it's fashionable to imagine automating the whole process, but everything I've seen is that the only systems that succeed are the ones that are augmenting an expert.
advaith08 5 hours ago [-]
yeah, the tricky thing about the experimental loop is that:
1. its very difficult to do it in a reproducible manner (the same experiment done twice often gives different results due to small undocumented changes)
2. its expensive to do at scale.
Both of these properties make it hard to hill climb on experiment. What's worked for us so far is precisely what you said - having human experts review and provide feedback. we distil their reviews into rubrics, and have LLMs act as proxy experts using these rubrics. We expect the models will hill climb using this approach, and will reach (close to) human expert level by doing this.
alansaber 1 days ago [-]
"Fewer iterations for materials science discovery" is a good spin. Closing the computational>experimental loop is the main challenge. This is the focus of my past research group, there is definitely potential, best of luck!! I have a crap write-up on this in case it's of interest https://alanyahya.com/writing/automated-materials-design
advaith08 1 days ago [-]
Cool read, and agree that closing the computation > experimental loop is key!
9 hours ago [-]
SpaceCoreDev 23 hours ago [-]
The "Claude's propensity to reward hack" line is the interesting part to me. We run a small system where AI agents (scripts, LLMs) act as the actual players in a persistent simulation, and reward-hacking-style behavior shows up constantly once an agent is left running unsupervised for a long time - it finds the shortest path to whatever metric you exposed, not the path you intended. Curious whether you've found any mitigation beyond just watching for it after the fact, e.g. changing what you expose as the optimization target versus what you actually want.
imko_ 19 hours ago [-]
I wonder what this would look like here. Seems like a space where keeping the exposed metric and the optimization target apart would be quite difficult.
Also curious: by what reasoning path do models typically end up reward hacking?
advaith08 20 hours ago [-]
yeah we were surprised by how much it does it. Our approach has been retroactive - we monitor the thinking trace, spot reward hacking behavior and then fix things.
We haven't faced this issue with Sol though - its been much more well behaved
areoform 18 minutes ago [-]
How do you have access to the thinking trace?
Melatonic 16 hours ago [-]
Love your idea of HBM on top of the chip. I've been saying for years that limiting memory to the edges is going to be an inherent physical limiter. I think the only reason it hasn't become more of a thing is that chip size has also been increasing with increasing compute.
What about HBM on the back side of the chip ? Essentially I'm thinking like a soldered on piece or another "socket" with pins like a CPU in the back of the motherboard. We rare see cooling elements there already and while most rack mount cases are not at all designed for more space there I don't see why it couldn't be a thing. Especially with liquid cooling.
Also makes me think of possibly using Gallium Nitride instead of silicone for surface levels elements ? Or maybe even an interface layer on top or bottom of the chip designed to be able to transmit power with less heat. Maybe that could be sandwiched between the chip and HBM.
akashramdas 16 hours ago [-]
oh that's a very interesting question! -- HBM on the back side of chips has advantage and disadvantges. Logic atop HBM means power delivery becomes the bottleneck. This is equally problematic. My understanding is that the industry seems to prefer HBM on logic currently. Also for the bonding, I think we are going to see hybrid-bonding very soon! Instead of solder, we can directly connect the indivdual dies.
I think the switch to GaN is already happening for power, but Si is considered better for logic dies like the GPU. Also GaN is worse thermally than Si so the problems are actually amplified
dhchun1203 20 hours ago [-]
the 8 hours vs 2 weeks framing is the part i'd want more on. generating
candidates got cheap, checking them didn't. what does the funnel actually look
like ,of the candidates from an 8 hour run, how many make it to synthesis?
asking because i hit the same shape in a much dumber domain and what got me was that the failures were quiet. nothing errored, output looked normal, it was just
wrong in a way only someone who knew the domain would catch.
akashramdas 19 hours ago [-]
You can see from our the benchmark that only one of the candidates proposed was determined to be worth synthesizing. Each individual candidate generation is quick, 8 hours is required for the model to iterate with various tools to find ones worth submitting.
We found that speaking to domain experts was critical in desigining a rubric that could catch these silent synthesis recipe failures, before we attempt the longer 2 week synthesis effort.
rytill 24 hours ago [-]
What required expenditures does a company like yours have on lab equipment / software, if any, to validate material properties?
advaith08 21 hours ago [-]
We're still figuring this out. We'll need some synthesis equipment (think CVD, PVD etc) and characterization (XRD, Raman spectroscopy) tools in-house to validate that we're making the right materials. We're considering developing these tools in-house - the models sometimes come up with clever modifications to them so that they can deposit new materials. We think equipment is as central to new material discovery as the material itself, and will probably need to be rethought to allow for high-speed AI based experimentation
Melatonic 17 hours ago [-]
What about aperiodic materials ? Does that go beyond (computationally) normal requirements ?
akashramdas 16 hours ago [-]
Aperiodic materials can also be studied. The number of atoms and cell size of what you want to simulate is more the limiting requirement computationally. Aperiodic materials weren't in this benchmark. Generally amorphous materials and polymers, with a few exceptions are < 2-3 W/(m K) in thermal conductivity. So we left them out of this study.
Melatonic 16 hours ago [-]
I meant computationally as in are they much harder to predict and simulate ?
Was specifically thinking of quasicrystalline materials and not amorphous. I know some of them have very unconventional properties so I figured they might be useful here. I don't think any of them are considered polymers but I could be wrong.
Wouldn't they require a totally different type of algorithm given that they often contain both a large number of atoms and odd cell size ? And the more dimensionally complex maths
advaith08 6 hours ago [-]
interesting! we haven't really considered these yet. However, we may soon have to - the recurring feedback we hear from industry is that crystallinity is a pipe dream, and that most materials are going to be amorphous (or perhaps quasicrystalline). MLIPs have lowered the computation cost for a large number of atoms/odd cell size, but it may be a while before they're accurate enough to simulate these scenarios
19 hours ago [-]
krtk00 1 days ago [-]
how do you measure the success/potential of a novel material/direction suggested by the agents? given you have limited time & resources - shortlisting the approaches for the synthesis stage becomes equally important as the approach itself.
advaith08 1 days ago [-]
There’s a variety of computational techniques that help us establish some confidence on the materials. Atomistic simulations can estimate stability and bulk properties of a new material, and we have synthesis experts (min qualification: PhD in thin film deposition) come up with rubrics on how to judge if a material/synthesis recipe is worth trying. All these approaches have known limitations, and improving them is the bulk of our work as a company!
There’s also a lot of work to be done in figuring out the minimal set of experiments required to know if a research direction/material set is worth pursuing
thanks for sharing! We think the Experiments-as-code path is not the right approach. the beauty of LLMs is their ability to ingest and reason over unstructured data - they remove the need for formalizing experiments. We tried using declarative templates to document our experiments, but realized that most of the interesting insights (for example, how viscous a liquid feels) is easier described by ranting about the experiment to a LLM, than formalizing it via constructs/code.
— GPT-5.6 Terra, reasoning summary, mid-run
This is hilarious
I think this is the first one that has actually taken the pain to say how many of the discovered materials are actually feasible which is a real step in the right direction. Probably worth keeping in mind the step beyond plausible synthesis which is the actual cost/effort of the material. There's not much point if you find out RuO2 would be better than SiO2, as an example, if Ru is orders of magnitude more expensive.
A challenge I think you'll run into is that I expect the biggest companies (e.g. IBM) will already be doing the part they need themselves. I heard tell of IBM in particular using ML to improve their own chips before LLMs came along, so I'd be shocked if these bigger companies weren't already doing this for their own problems. Also, if you aren't doing the experiments yourself, it's always going to be a challenge to find a partner to test things for you and this will probably be the major time sink.
There are domains (or, mems, photonic sensing, high power) where innovation has a bit more flexibility for the early going.
As an aside, the "Fable lies and cheats" section made me laugh -- have encountered this same failure mode, albeit for much simpler models. Polymerization (I realize this is not exactly the same thing) is an unbounded 3D playing field for constraint escape. You can try to put an additional constraint on polymerization, but then it will just make minor variations on the monomers...
We did find there are ways we can prevent this particular reward hacking, just consistently surprised by how these models can find tricks to technically achieve the set goal
Once you have the experimental loop running, I suspect it will be quite difficult to hill climb on this task.
There will be some improvements you can make to the harness, but I suspect you'll be doing a lot of human in the loop review and providing feedback that goes back into the harness instructions.
I know it's fashionable to imagine automating the whole process, but everything I've seen is that the only systems that succeed are the ones that are augmenting an expert.
Both of these properties make it hard to hill climb on experiment. What's worked for us so far is precisely what you said - having human experts review and provide feedback. we distil their reviews into rubrics, and have LLMs act as proxy experts using these rubrics. We expect the models will hill climb using this approach, and will reach (close to) human expert level by doing this.
Also curious: by what reasoning path do models typically end up reward hacking?
What about HBM on the back side of the chip ? Essentially I'm thinking like a soldered on piece or another "socket" with pins like a CPU in the back of the motherboard. We rare see cooling elements there already and while most rack mount cases are not at all designed for more space there I don't see why it couldn't be a thing. Especially with liquid cooling.
Also makes me think of possibly using Gallium Nitride instead of silicone for surface levels elements ? Or maybe even an interface layer on top or bottom of the chip designed to be able to transmit power with less heat. Maybe that could be sandwiched between the chip and HBM.
I think the switch to GaN is already happening for power, but Si is considered better for logic dies like the GPU. Also GaN is worse thermally than Si so the problems are actually amplified
asking because i hit the same shape in a much dumber domain and what got me was that the failures were quiet. nothing errored, output looked normal, it was just wrong in a way only someone who knew the domain would catch.
We found that speaking to domain experts was critical in desigining a rubric that could catch these silent synthesis recipe failures, before we attempt the longer 2 week synthesis effort.
Was specifically thinking of quasicrystalline materials and not amorphous. I know some of them have very unconventional properties so I figured they might be useful here. I don't think any of them are considered polymers but I could be wrong.
For example (no personal connection):
https://arxiv.org/abs/2409.07735
Wouldn't they require a totally different type of algorithm given that they often contain both a large number of atoms and odd cell size ? And the more dimensionally complex maths
Have you seen
https://arxiv.org/pdf/2605.04375
https://arxiv.org/abs/2307.07522