this is the kinda debugging case i find rlly annoying/

everything says success.

no exceptions no failed tool calls. no obvious timeout the workflow completes but the final result is still wrong

when that happens, what’s your first move?

do you guys usually:

• start from the final output and work backward
• compare against a previous good run
• inspect state transitions
• check retrieval/tool behavior
• look at model inputs
• replay it
• check business state outside the trace
• just read the whole thing until something looks off

interested in what people actually do in production not the idealized version but thats fine too. and if you have anything you've built to help with this process I'd love to see it :)

I was wondering why dynamic competitive games like Rocket League don't have a decision-quality engine like chess has with Stockfish. Something that doesn't just measure your boost usage or speed, but actually evaluates whether your positioning or challenge was the mathematically optimal play.

I initially drafted a naive proposal based on slicing replays into 5-second physical rollouts in the cloud to test decision trees. However, after using AI models to strictly critique the physics and compute feasibility, Machine Learning seems to offer a much more viable path:

Decision Quality: Evaluates plays using Offline RL (Trajectory Transformers & Implicit Q-Learning) over a Sequential POMDP. Instead of chaotic physics simulations, it calculates decision probability based on pro dataset distributions.

Anti-Cheat & Smurf Detection: Uses frequency spectrum analysis (FFT) on input signals and kinematic limits (4th derivative/Snap) to catch bot scripts, along with NLL divergence to mathematically flag smurfs.

Disclaimer: I want to be 100% clear that I am not looking for any personal credit or clout for this. As a 1st-semester student, I don't even have the technical background to understand or build the advanced machine learning model proposed by the AI. I just thought this could be fascinating and wanted to share it with the community in case it sparks interesting discussions or research for the future.

You can read the full breakdown here: https://github.com/Pnlw/esports-performance-evaluation

So I've been messinga round with embedding models for a bit, and I think they are interesting enough to experiment with. They are useful for rag, especially in a localllm sense because you can ground your answers in truth.

But what happens if you have a billion documents, and you decide to upgrade your model to a "better" one? on an h100, that would take about 108 days, just to upgrade the vectors so u can start serving again (tested qwen embed 8b on h100). Even if you aren't doing 1b vectors, and are doing just 50 million, upgrading can still take a considerable time.

Me and my research lab decided to tackle this problem, and we came up with embedflow.

The method is really simple; from the old index made with the source model, take K documents and rerank them with the new model. We see that when K is sufficient, the retrieval quality is the same as target model. (determining k is the hard part). I've tested 63 migrations on upto 1 million documents.

The best result I got was upgrading qwen4b -> to 8b, and at 50 documents, it was the same as native retrieval.

This method forgos the expensive backfill that comes with upgrading, as you can directly take documents from the old index.

embedflow works with qdrant, and can be easily downloaded with pypi

pip install embedflow

the github is public: https://github.com/arnsri33/embedflow

I want you guys to try it out, and see if you guys can use it in your own workflow.

I'm testing a new approach for reducing the cost of image-based LLM inference.

I evaluated it on the MOMA Graph benchmark, using 1,315 questions. Compared with using GPT-4o to process the original images directly, I observed approximately:

• ~95% lower token usage
• roughly the same accuracy as the GPT-4o direct-image baseline

I'm intentionally not sharing implementation details yet because the method is still under development.

I'm mainly trying to understand how strong the result itself is.

If these numbers hold across larger and more diverse benchmarks, would you consider this a meaningful result in multimodal AI efficiency?

What evidence would you want to see before taking the claim seriously?

For example:

• more datasets
• stronger baselines
• statistical significance
• latency measurements
• API cost comparison
• performance across different models
• failure-case analysis

I'm especially interested in feedback from people working on multimodal models, VLM efficiency, or inference optimization.

A few weeks ago, I saw this post where the author trained a SIREN MLP to implicitly memorize Bad Apple as a coordinate function: (t, y, x) to pixel.

That got me curious about a slightly different formulation: instead of handing the network a timestamp t, could a small recurrent dynamical system (RNN-ish) learn the continuous temporal flow in latent space and generate the entire ~6,500-frame full resolution video autonomously from a single initial condition (h_0, c_0)?

The code, weights, and analysis tools with rollout scripts, plots, and standalone models are shared here: GitHub: SEBADA321/BadAppleRNN.

Architecture & Inference Footprint

At inference time, the system receives no timestamp inputs and evaluates in a closed loop:

(h_t, c_t) -> Recurrent Transition (CTF) -> (h_{t+1}, c_{t+1})
                   |
                  h_t -> Frame Decoder -> 384x512 Grayscale Frame

• Latent Dimension: 64-D for h_t (decoded) and 64-D for c_t (internal memory manifold to separate visually similar frames at different timestamps).
• Recurrent Transition (ctf): 4-gate LSTM-style recurrence with orthogonal initialization (16,640 parameters, 65 KB).
• Frame Decoder (fd): 4-stage bilinear upsampling with depthwise-separable convolutions (400,361 parameters, 1.56 MB).
• Initial State: A single pair of 64-dim vectors (h_0, c_0) (128 floats, 0.5 KB).
• Total Inference Model: 417,129 parameters (~1.60 MB in FP32).
• Runtime Performance: >200 FPS on an RTX 4080, with ~17.2 MB peak active VRAM during decoding.

Training an autonomous system across ~6,573 steps from t=0 directly was probably computationally unstable due to vanishing/exploding gradients and compounding error. The training setup I was circling around used several targeted techniques:

• Learned Latent Teacher Tables: During training, I optimize a pair of tables h_table[t] and c_table[t] alongside the model. This allows parallel segment training starting at arbitrary timestamps over a finite horizon K. These tables are scaffolding and are discarded entirely at inference.
• Rollout Horizon Curriculum: I started training with K = 2 and progressively doubled the rollout length (K = 2 -> 4 -> 8 -> 16 -> 32 -> 64 -> 128 -> 256 -> 512). Each horizon doubling produced a characteristic jump in loss before the transition function adapted to the longer trajectory.
• State Perturbation Noise (sigma = 0.005): To prevent the model from learning a brittle 1D line that diverges under small numerical errors, Gaussian noise was added to the state before passing it into the transition function (z_hat_{t+1} = F(z_t + epsilon)), while evaluating the loss against the clean target. This encourages the recurrent map to contract small deviations back toward the trajectory.
• Second-Difference Acceleration Regularization: Penalizing velocity (||h_{t+1} - h_t||) risks collapsing the trajectory. Instead, I penalized discrete acceleration (jitter) via second differences: ||h_{t+2} - 2h_{t+1} + h_t||_2^2. This enforces smooth trajectories without penalizing motion.
• Optimizer & Momentum Management: I used AdamW (1x10^-5) for the decoder/tables and Muon (0.005, momentum 0.95) for the recurrent weights. To prevent accumulated momentum from acting as stale inertia when K doubled, momentum buffers were scaled by 0.2 every 10 epochs starting from the epoch 500.
• Chunked Decoding: To handle long horizons at K = 512 without overflowing VRAM during training, the decoder was evaluated in temporal chunks of 32 frames.

Some interesting things

1. The model could unroll the full 6.5k sequence even if it was, technically, trained on up to 512 frames. So that was a success.
2. Training loss vs. autonomous rollout: The checkpoint with the lowest numerical training loss was not necessarily the best at autonomous generation. Because K changes across the curriculum, raw training losses are not directly comparable across stages, and short-horizon teacher-forced agreement does not guarantee long-horizon stability.
3. Dynamical stability over parameter scale: The challenge was not increasing parameter count (the recurrent transition is only 16k params), but conditioning the dynamics through noise injection and acceleration penalties so error doesn't compound over thousands of recurrent steps.
4. I need to improve the decoder a lot, I was mostly focused in getting the recurrent part right, and training was slow. Now that I gave gotten a successful result I will focus more into optimizing the CNN decoder.
5. There are no skip connections nor normalization, which was interesting too. Obviously no attention either since I wanted to keep it simple.I also didn't want to use Truncated BPTT.

Not completely scientific, since I was doing some changes mid run or many at once, which makes it kinda not clear what contributed more. I used 'AI' to help with writting the post and README. Part of the code was also generated that way, but the architecture is what I came up with on my own and from a previous project too. There are probably many parts to improve too, so glad to get some feedback!

I used an LLM to iteratively evolve an optimization algorithm rather than solve the packing directly. Starting from a simple seed solver, the LLM proposes algorithmic changes guided by a scoreboard of results and a history of prior attempts, and each candidate is scored by an independent verifier so improvements are kept and failures discarded. On the Packomania csqv benchmark it improved the best-known sum-of-radii for 10 values of N from 101 to 114, by 2.4 to 5.4%, in 15 iterations. Total LLM cost was $27.72. Packomania accepted the results independently.

Paper: arxiv.org/abs/2609.05093

Code + solutions: github.com/ucsandman/discovery-loop

Benchmark: packomania.com/csqv/csqv.html

Happy to discuss the plateau-detection stopping rule, that's the piece I'd most want critique on.

Background : Our T-PAMI submission was rejected despite receiving three highly favorable reviews. The AE inadvertently revealed that the decision relied on negative comments attributed to a “fourth reviewer.” However, the actual fourth reviewer had submitted a positive review, which subsequently disappeared from the review record under the AE’s handling. We have spent the past six months pursuing this matter with IEEE. (Original post: [How to get rejected by IEEE T-PAMI with 'Excellent' scores?[D])

Latest update : A few days ago, following an investigation by the IEEE Computer Society Committee on Integrity, the T-PAMI Editor-in-Chief formally acknowledged that four reviews had in fact been received, thereby confirming the existence of the missing fourth review.

Hi everyone! We just released Rustuna (GitHub: https://github.com/optuna/rustuna/ ), a high-speed, memory-efficient implementation of Optuna built in Rust.

• Optuna-Compatible Design: Keeps the familiar API and concept of Optuna.
• Zero Python Dependencies: Mitigating the risk of supply chain attacks.
• Lower Memory Footprint: Optimized memory management natively in Rust.

For details, please check out the following blog post.

https://medium.com/optuna/announcing-rustuna-cc82a6815bf7

Our research team has been exploring an alternative approach to achieving interactivity and better responsiveness with LLM systems.

One of the team members wrote up a post about it:
https://research.yandex.com/blog/the-kv-cache-as-an-agent-runtime

The post sums up the overall idea of modifying models inference state (KV-cache) for achieving a more interactive LLMs. This idea was used in our lab's previous papers Hogwild! Inference, and AsyncReasoning, the post also contains a preview of the future work in this direction, where a Qwen3.8-27B agent is playing a DOOM env interactively using similar techniques.

We think that its interesting whether model inference/runtime design is itself an under-explored axis of agent capabilities, alongside models and the harness (e.g. harness is too abstract, changing model is too costly, do we need something in between?)

Hello all,

I'm a radar signal processing engineer and i trained a 5-class classifier (car, large_vehicle, two_wheeler, pedestrian, pedestrian_group) on RadarScenes radar point clouds.

The input vector is a per-scan histogram (16 bins) and the network is a 3-layer MLP. The loss function is a class-weighted cross-entropy loss. This work is based on "Histogram-based Deep Learning for Automotive Radar" paper.

I scoped the project to be one scan only. Accumulation of multiple scans is the next step.

Data

Class Imbalance: two-wheelers and large_vehicles has a low number of occurences.

Aggregated Classes: two_wheeler mixes bicycles and motorized variants; large_vehicle merges trucks, buses, and trains together due to data scarcity.

Sequence Bias: Long tracks of slow-moving objects can skew a particular data split velocity distribution, causing high F1 score variance across folds.

Ablation studies

I tried with bigger MLPs, alternative feature encodings, and different histogram binning, all moved performance less than the variation caused by changing the train/validation/test split. I measured that split sensitivity across 6 folds, keeping the same proportions.

Changing the histogram to per-instance statistics (mean/median/std) slightly degraded performance.

Main findings

Macro F1 rises from 0.381 to 0.764 as the naturally occurring number of radar detections per instance increases from 1 to 5. I trained the model normally using all available detections, then bucketed its existing validation predictions by each instance's detection count and computed macro F1 per bucket.

The classes car and pedestrian has the best performance and two_wheeler has the worst.

A car is often confused as large vehicle when the car was wider than usual or had a unusually high rcs (which can happen due to multipath for example).

The two_wheeler is often confused as pedestrian because their vr_compensated distributions overlap, which is the the model's single most important feature for these two classes. A stationary or idling two_wheeler is indistinguishable from a pedestrian.

I uploaded an image with ground truth vs predictions: A nearly stationary two-wheeler which contains a single point was predicted as pedestrian, because its velocity is near zero, indistinguishable from a pedestrian. A car in the same scene, also with just one point, is classified correctly, since RCS and Doppler are enough for that class.

Full writeup here: https://github.com/brunopinto900/radar-ml-autonomous-driving/blob/main/MLP_Report.md

Future work

Implement other spatial encoding schemas (point net for example) and accumulate multiple scans to tackle the challenge of sparsity and explore the concept of micro-doppler.

Is LfD and BC research being effected by recent advances in (so-called) Frontier LLMs? Or is research in LfD and BC sort of going along in an independent direction from these?

Are you seeing any use from ViTs or VLAs?

Any other recent advances you would like to bring up?

One thing that has bothered me about LLM benchmarks for a while is that most of them are essentially snapshots.

A model is evaluated, a score is published, and we tend to talk about that score as if it describes a relatively stable object. But with API-served models, the thing behind the model name can change over time: serving infrastructure changes, provider configurations change, versions change, and sometimes behaviour changes without an obvious public version transition.

So we started approaching benchmarking as a longitudinal measurement problem rather than a leaderboard problem.

We continuously evaluate models across coding, multi-turn reasoning and tool use, while also running lightweight probes at a higher frequency. The important part for us is not simply asking "which model scores highest?", but:

• Is the model behaving differently from its own previous baseline?
• Is the change larger than its normal repeated-call variability?
• Did the benchmark configuration itself change?
• Is the effect concentrated in a particular task?
• Is it correlated across models from the same provider?
• Is an apparent degradation actually an availability/infrastructure issue rather than a capability change?

One historical analysis covered 31,352 repeated score observations across 49 models. The standard deviation of within-day scores was 2.80 points, while the standard deviation of between-day daily medians was 8.43 points.

That is roughly a 3:1 difference.

I don't think this result by itself establishes that providers are changing models day-to-day - there are too many possible confounders for that conclusion. Task composition, sampling, missingness, provider behaviour and methodology changes all matter. But it was enough to convince us that temporal variation deserves to be measured rather than treated as noise around a permanent leaderboard score.

Our current approach therefore keeps benchmark configurations versioned and only compares longitudinal observations produced under compatible measurement conditions. We use repeated execution-based evaluation where possible rather than an LLM judge, keep availability failures separate from valid task outcomes, track serving/version metadata when providers expose it, and run change detection over the resulting time series.

Another problem we're increasingly interested in is benchmark recognition and contamination. Once a benchmark becomes sufficiently visible, publishing every live task, prompt transformation and hidden test potentially changes the thing you're trying to measure. For that reason we've tried to separate methodological transparency from publishing the entire live evaluation set.

We've now written up a public version of the methodology. It intentionally explains the measurement design, assumptions, limitations and statistical interpretation, while withholding the exact live task bank and some operational parameters.

PDF: https://aistupidlevel.info/asl-public-benchmark-methodology-2026.pdf

I'm particularly interested in criticism from people working on evaluation, change-point detection or production ML.

A few questions I'd genuinely like opinions on:

1. For longitudinal LLM evaluation, would you use daily medians as the primary time-series unit, or model the individual repeated observations directly?
2. How would you distinguish genuine model drift from provider/infrastructure effects when version metadata is incomplete?
3. How much of a live benchmark should remain hidden to reduce contamination while still making the methodology scientifically inspectable?
4. Are there better approaches than change-point detectors for this kind of non-stationary, relatively noisy model-performance series?

Disclosure: I'm the founder of AI Stupid Level, the platform that produced these measurements. The purpose of posting this here is to get technical criticism of the methodology rather than promote the commercial product.

When I first started working in scientific machine learning, I understood the physics much better than the coding. Every time I wanted to try a new physics-informed neural network problem, I had to start almost from scratch: changing the PDE, updating boundary conditions, modifying the architecture, tweaking the training schedule, debugging errors, and generating plots—all by hand.

That frustration pushed me to build PINNStudio. It is a free, open-source no-code GUI designed to eliminate boilerplate code so you can focus entirely on the physics.

Instead of rewriting a new script for every problem, you can define your setup directly through the interface:

• PDE Definitions & coupled multi-output PDE systems
• 1D or 2D domains with boundary and initial conditions
• Network architecture & custom training schedules
• Forward problems (solving known PDEs) or Inverse problems (estimating unknown parameters from data)

What happens next?
PINNStudio automatically generates the code (built on top of DeepXDE), runs the model, streams the training log, and displays live loss curves and solution plots directly inside the app. It also includes built-in templates for classic equations like Heat, Allen-Cahn, and Cahn-Hilliard.

• GitHub (Open Source): https://github.com/AsfandyarKhan72/PINNStudio
• Quick Install: pip install pinnstudio

My hope is that this will be helpful for students and researchers with limited coding experience, as well as experienced PINN users who just want a faster workflow.

I’d love to get your feedback, feature suggestions, or bug reports! Huge thanks to Lu Lu and the DeepXDE team for creating the foundation that made this possible.

I feel that reproducibility is now a lost cause in machine learning research for three reasons:

1. Many research is moving towards the physical AI territory, where you need expensive hardwares or even entire laboratories with high-speed cameras, in order to perform an experiment. You truly have no idea if the experiment can be reproduced and have to trust the demo. But demos are not perfectly reliable. Plus people are incentivized to only show the part of the demo that works. The entire system can fall apart the moment the recording stops.

2. You have big AI companies releasing various tools, which they claim to solve a host of problems with certain amount of accuracy or efficiency. Unless you work at those companies there is really no proof of that and you will have to take their words on it. They have strong financial incentive to blow-up those figures. There is no solid way to check it either because the problem that they solve are so vague and subjective.

3. We need to address the elephant in the room which is that people are incentivized to produce non-reproducible work to prevent their lunch being eaten by their competitors or looking bad. That's why some of us will probably never get a reply when we email the authors for their code.

So what now? Maybe everything will be OK because we can contrast it with scientific progress in earlier parts of history, e.g., building the atomic bomb or sending people to the moon. These projects had low "outside reproducibility" but high "internal reproducibility". Plus all these work were mathematical in nature and carefully checked. But I don't think many areas of machine learning research is like that. What do you think? Should reproducibility be abandoned? If not how is it best implemented going forward?

AACL-IJCNLP 2026 acceptance results will be released in a few hours.

Feel free to share your thoughts and feelings! How did you do?

building a missing data infrastructure and started benchmarking long multi-session conversations (LoCoMo). I know the data looks like: people, facts, claims, events, timestamps, relations. So I extract those into a graph.
I did not look at the QA pairs while building extractors or retrieval rules. No “if question contains X, fetch fact #173.”
Recall is very high and it keeps working on new conversations in the same format.
Is this classical overfitting, or just schema-aware engineering? What is the cleanest test that would convince you it isn’t leakage.

Hi All,

Was reading AIStats' website and it seems like abstract submission is due in 3 weeks.

Does anyone know where to find the LaTex template for 2027? It seems like very little information is available on their website.

Another question, is a Quant Finance paper a better fit for AIStats or ICLR?

Some background about the paper:

• Rejected by UAI with 76654, had some errors with proofs had to fix it by re-writing 9 pages during rebuttal. AC rejected the paper saying the changes were too substantial and unable to be fully verified during rebuttal period.
• Resubmitted the fixed paper to a finance conference, won best paper award (best paper for this conference usually end up in journals like JQFA, which is just 1 tier below the big 3 in finance), had the chief editors of a Q1 finance/math journal in the conference verbally offering he will take this paper if we submit it to his journal. Unfortunatley my department requires at least 1 Comp Sci paper to graduate, so my plan is to try and get this paper accepted into a Comp Sci conference, then submit an extension to that Q1 Finance/Math journal.
• Rejected again at ICDM, despite having all positive scores. Our AC meta-review was blank so we still do not know why we were rejected. All of our emails receieved no reply.

I am torn between ICLR or AIStats to re-submit this paper to. My worries are:

• In comp sci venues we frequently get comments like "this paper lacks novelty. The method is just XXXXX, the math is just XXXXX."
• But I had a scroll through at previous year's AIStats papers for key words like finance and there were none. It seems like AIStats is very pure stats, not that applied. My co-author is worried that the math in our paper is not hardcore enough.

We have never submitted to neither venues in the past. Would be nice to get some advice.

I ran a side-by-side ML text-processing and model-training workflow using Fable 5.1 vs. Astra (both on xhigh), and the results could not have been more different. Warning, long post.

TL;DR -- Astra codes more agentically, Fable more coherently. Fable writes better and follows directions better. Astra's final outcome was slightly better, and its scientific rigor/reproducibility was noticeably stronger. Both models improved their F1/Accuracy by 0.02-04 after human feedback on their approach, demonstrating that neither have mastered the AI/ML text processsing, vectorization, and model training process completely.

Astra is a better coder, writing a stricter evaluation protocol (70/15/15 train/val/test vs. Fable's basic 80/20) that selected its model using a held-out validation set vs. Fable's simpler test F1-based selection. It also debugged more deeply, as both models hit a gensim 4.4 compiled-kernel bug: Fable tried to figure it out, failed, and just hid the stderr notices on affected runs (though told me it had done so), while Astra root-caused it aggressively, then fixed the environment by downgrading gensim alongiside compatible NumPy/SciPy dependencies.

Astra wrote hardened training-run.py code the forced the uv venv it rebuilt without changing my default one, SHA-256'd the corpus to ensure reproducibility on later runs, output a split manifest and run-summary.json, and rendered a headless browser for QA with screenshots (not sure this was necessary, but impressive overkill all around). Fable's builder script was ephemeral, living only in tmp, and less intense overall.

Astra deployed subagents more effectively, making use of my pre-built notebook-reviewer and citation-checker agents, the former of which caught a real bug via review (sentence-final word-loss tokenization defect) and fixed it, retaining a regression test in the process. Fable overlooked this issue because, for some reason, it did not call the subagents I had available (which is surprising, usually it's pretty good about this).

If you're looking for an agent to autonomously grind through a broken environment, leaving a forensic audit trail, that's Astra. However, this review isn't over yet, and Fable is about to make a comeback.

Astra confidently shipped a significant verifiable text encoding defect. Working with UTF-8 data, Astra insisted Windows-1252 decoding preserves currency symbols, but the final HTML output shows mojibake throughout where currency symbols were in the original data. Fable read UTF-8, verified it, and rolled with the boring default for correct output.

I also had both models draft an analysis report for the run, and Fable's was significantly more insightful. As much as I hate Claude's recognizable writing style, a) 5.1 has toned down the Claudeisms significantly, and b) Fable went above and beyond my grading rubric, running an ablation on different parts of the text pre-processing pipeline to surface an expensive step that does basically nothing, and noting a discrepancy in the classification ranking based on a complexity I'd have overlooked. For writing prose, I'd pick Fable 5.1 any day, and I haven't said that about Claude in a while.

Speaking of writing, Fable writes code that is more idiomatic and readable. It definitely resembles more what I would write than what an LLM would choose to write without constraints (and yes, I had a whole coding-conventions.md document that applied my requirements to both models, Fable just followed it better and writes more naturally to start with). There were some parts of Astra's code where I had to squint really hard to figure out what was going on, and why. This matters to me because I'm not the strongest coder (still trying to get better), and I need to understand the code to learn from it.

Finally, Fable scoped its work better: It spent its time and tokens doing repeated runs, tweaking hyperparamters and retraining the models to find the optimal settings while Astra deeply debugged the gensim error. It found significant uplift through this process, though that only allowed it to roughly match Astra's numbers (see table below). Astra seemed to hit a home run right off the bat with its training process, so I don't know if it would have executed the same workflow or not. Astra also mutated my venv by adding PyTorch, when I built it a certain way to force the models to use TensorFlow+Keras for more concise code, then reversed course and went with TF anyways in the end. The models finished in roughly the same amount of wall-clock time.

Here are the final results, with one minor caveat -- Astra's test set scores exceed its val set, so it might have drawn a lucky test set that increases its score artificially (the pipeline has no leaks or data quality issues for either model, however):

Best Logistic Regression and LSTM for each model, ranked by macro F1:

I do want to note that these final scores were after I provided both models identical feedback on common pitfalls of the text data cleaning, vectorization, and model training process once their initial runs were complete. Both models improved by a similar amount (0.02-0.04 F1 and Accuracy) from that generic guidance (not tailored at all to either's specific shortcomings or step of the process). That was the only intervention in otherwise autonomous work, and it was just because I wanted to see if they could learn to improve their approaches with additional context on optimal methodology, which they both did to similar degrees.

I hope this post offers a little bit of help in some way for folks wondering how either model stacks up for real work, particularly if you're an AI/ML student like me.

A researcher has reported a jailbreak of GPT-6 Astra within a day after release.

The attack is described as combination of TIP (Task-in-Prompt) attack from ACL 2025 paper with four other unnamed techniques.

TIP attacks exploit the model’s reasoning/instruction-following behaviour by hidding the harmful objective inside another task, like solving a cipher or executing a Python code. For GPT-6, the researcher says the original minimal TIP attack was no longer sufficient and had to be reworked.

They have reportedly disclosed the details privately to OpenAI rather than publishing the jailbreak.

The same researcher reported jailbreaking GPT-5 within an hour of its release a year ago.

Source: screenshot/post from the researcher; their ACL 2025 TIP paper linked in the original post.

Just received an email about the automatic reference/citation checker. Did anyone receive a follow up email about whether the checker was included in the paper's decision making too, along with the general instructional email?

Abstract

Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant? To this end, we introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes: <global> (full context), <focus> (a specific region), and <local> (recent output only). The inference engine parses these declarations like tool calls and skips most of the KV cache read. Under zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) significantly reduces total attended tokens during decoding (52.0%, 31.1%) with modest accuracy drops (1.27pp, 2.75pp) that shrink with model scale. DA unlocks a new axis of sparse attention, with further potential under training-based methods that future work can explore.
Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as:
arXiv:2609.02737 [cs.CL]
 
(or arXiv:2609.02737v1 [cs.CL] for this version)
 
https://doi.org/10.48550/arXiv.2609.02737
Focus to learn more

From what I've seen online so far, the description of these systems is roughly:

They asked the model (often Aster) to generate statements in LEAN and then submit those to a LEAN compiler to be checked. Based on the results of attempting the LEAN compilation, they somehow add those statements as fact. When the full proof in LEAN compiles, the system is finished.

I can imagine trying to jam as much of a proof as possible into the context window but some of the papers these systems have produced are hundreds of pages. To me this would indicate that somehow the paper is being built piece by piece and being assembled before being submitted to LEAN. This resonates with the part of my understanding that after checking LEAN compilation there's some kind of management of "facts."

I would like to try to implement my own janky version and see if it can answer a question I have about higher dimensional geometry. I'm struggling to find a meaningful way to compose larger ideas from smaller ones. I can imagine it's relatively simple if you know what to do.

What things have you seen? Do you have any ideas you haven't seen that might be interesting to try? Is this a fool's errand because you really need huge amounts of hardware to do anything meaningful? I would welcome any thoughts or links on the matter, cheers

an observation : GPT-5-class models are genuinely capable(They are) of doing a substantial fraction of knowledge work, why haven’t we seen a noticeable productivity shock in the real economy yet? Is AI actually less economically useful than the benchmarks suggest—or are organizations simply too slow, constrained, and inefficient to turn model capability into measurable output?

Are we confusing “AI can do the task” with “AI can replace the economic system built around the task”? If GPT-5 is already this capable, what exactly is the bottleneck preventing that capability from showing up in GDP and productivity statistics?

My take :

There is no question that these models are genuinely impressive. The question is whether that intelligence is actually translating into measurable economic productivity.

People are already asking whether models like GPT-6 or equivalent. Claude, and Gemini will replace large sections of white-collar workers. I think there is a much simpler question we should ask first: if these models are already so capable (to me they definitely are capable enough)

at a huge range of knowledge work, why haven't we seen a correspondingly obvious increase in productivity?

I'm not even talking about GPT-6 or whatever comes next. It's probably too early to judge a newly released model. I'm talking about the current generation—GPT-5 and its equivalents from Google and Anthropic. These systems are genuinely good. They can write, summarize, analyze documents, explain technical concepts, generate code, reason through problems, conduct research, manipulate information and perform a remarkable range of tasks that previously required educated human labour.

And yet, looking at the world around us, something feels strange.

Where is the enormous productivity shock?

Why don't we see a dramatic effect on GDP growth? Why don't we see massive increases in output per knowledge worker? Why don't organizations appear to be accomplishing dramatically more with the same number of employees? Why does the broader economy still look remarkably similar to the pre-LLM economy?

Coding is probably the clearest exception, and even there the picture is complicated. AI can make programmers substantially more productive in certain tasks, but software development still involves architecture, debugging, verification, integration, requirements, security, deployment, maintenance and—most importantly—human judgment. The bottleneck often moves rather than disappears.

almost every knowledge profession, the gap between "the model can perform this task" and "the organization can therefore produce substantially more output" is different it seema.

A lawyer might be able to use an LLM to draft a document in minutes instead of an hour. But the lawyer still has to verify it, take responsibility for it, communicate with the client, comply with professional regulations and integrate it into an existing workflow. A doctor can use AI to summarize medical literature, but diagnosis and treatment remain embedded within a much larger institutional system. A researcher can generate dozens of hypotheses, but experiments still take time. A manager can produce reports instantly, but meetings, organizational politics and decision-making remain.

the possibility: perhaps the bottleneck is no longer intelligence.

Perhaps the bottleneck is everything surrounding intelligence.

Organizations, regulations, verification, trust, coordination, physical-world constraints, legacy software, incentives, management structures, liability and simply the fact that human institutions change much more slowly than technology.

This also makes me skeptical of simplistic claims that "AI can already do X, therefore everyone doing X will soon be unemployed."

Technical capability and economic substitution are not the same thing.

The internet could transmit information essentially for free, but that did not instantly eliminate newspapers, universities, governments or offices. Computers could perform calculations millions of times faster than humans, but most accountants and engineers did not disappear. Automation often increases the productivity of workers while simultaneously changing what their jobs consist of.

As with the major Grok release, Elon Musk said it is "as good as most top phds", my question after more than a year? (& he ain't wrong with the benchmarks), my question is, how many phds it has replaced in xai or spaceX?

did he stop hiring phds? if not, why?

So I find the current situation genuinely puzzling.

We have perhaps the most powerful general-purpose cognitive technology ever deployed, and yet the physical and economic world doesn't look radically different.

Maybe we're simply in the early stages and adoption takes years.

Maybe the productivity gains are real but are being absorbed into quality improvements rather than measured output.

Maybe GDP is simply a poor instrument for measuring the value created by AI.

Or perhaps current models, despite their extraordinary capabilities, still lack some crucial property required for autonomous economic production: reliability, persistence, agency, contextual understanding, verification, or the ability to operate continuously inside messy real-world systems.

idk which explanation is correct.

Benchmark scores:

https://openai.com/index/gpt-6-astra/

Above, GPT-6 uses a harness for ARC-AGI-3, and is at about 60% without one:

Prior to the launch, OpenAI President Greg Brockman said "I think it’s not unreasonable to feel that we are now in the AGI era".

GPT-6 is now joining a growing list of models that greatly exceed the human baseline on GDPval-AA v2:

If we have AGI, why do human knowledge/remote workers still have jobs? Is it just a matter of time until the economy replaces a large number of humans with LLMs, or are LLMs lacking something that these benchmarks fail to measure?