Differences

This shows you the differences between two versions of the page.

--- add-rebuttal [2026/06/30 14:42] – sam
+++ add-rebuttal [2026/07/01 14:44] (current) – removed sam
@@ Line 1: / Line 1: @@
-====== NL-Scenarios: Authoring Multi-Agent Traffic Scenarios in Natural Language ======
-**Venue:** VL/HCC '26 \\
-**Authors:** Anonymous
-===== Short Summary =====
-The authors present NL-Scenarios, an intermediate representation for authoring traffic scenarios, which allows traffic scenario authors to make small tweaks to a scenario without re-prompting a language model, or learning a more specific DSL for scenario specification.
-===== The Review =====
-Thank you to the authors for submitting this manuscript. I am happy to see work which augments language models with additional knobs for increased explainability and fine-grained adjustment. I found that the beginning sections of the paper read well and presented the system's design clearly. However, I am less convinced by the evaluation, which I think fixates too much on the technical behavior of the model the authors employed to drive the traffic simulation, and spends too little time trying to convey what kinds of edits or inspections are made possible by the authors' tool.
-In section III.B we see a brief example interaction where the user edits the velocity of one of the cars in the simulation. However, this action is on the JSON scenario representation, which precedes the graph, attention mechanism, and simulation. The paper mentions that "every artefact between the sentence and the 3D output is inspectable and editable." Since the paper's contribution is this graph intermediate representation, I would have expected to get a sense of the interactions that are possible //because of// this intermediate representation.
-For example, what if the wrong node is attended to in the graph view? What recourse would the user take in such a situation? Or, what if the vehicle's behavior isn't as expected in the simulation? What if the structure of the graph itself isn't as the user expects? Are these situations even possible? After reading the paper, I have little clue about these questions, yet I think they are important in understanding the tool in a human-centered way. In other words, I don't have a sense of what it would feel like to use the system, and the capabilities of the intermediate represenatation from a user's perspective.
-Concretely, I think that the authors should consider adding a broader range of usage scenario descriptions to establish evidence that the chosen intermediate representation is useful to users. My score reflects the fact that I believe these changes to be somewhat substantial.
-In the remainder of the review, my comments are arranged roughly by section, and contain more local low-level feedback.
-  * I don't consider it a major concern, but I think the title of the paper is a little misleading; the primary contribution is the intermediate representation, not the capability to author traffic scenarios using natural language, which has been explored already in prior work, as this paper cites in the introduction.
-  * The model trained to produce driving commands from the graph IR can account for different driving modes, such as "aggressive," but it's not clear to me where the source of truth for this accounting comes from. Does HighwayEnv understand what it means for a driver to be "aggressive?" I couldn't find that kind of information on their GitHub page, so I think it's worth mentioning in the main paper.
-  * Figure 2/3. I'm not sure that Figure 2 is necessary at all given it's depicting the same panel from Figure 3. If the authors want to keep both figures, however, I would recommend showing Figure 3 first since it gives context to the entire interface, before showing the detail view, especially since the figures are on different pages.
-  * In section IV.E, I'm not convinced of the explanation about why k!=5 would reuslt in reduced performance. How can we be sure that it's not, for example, due to a bais for graphs of a certain degree introduced during training? Furthermore I think that "visual signal" needs to be explained; is it referring to the thicker line employed when two nodes attend to one another?
-  * Figure 4 is showing four scenarios along with their prompts, and I understand the left of each frame to be a display from HighwayEnv, and the right to be from CARLA. However, I think it would be more interesting to show the graph representation of each scenario. Again, this would be in service of understanding the properties of the graph IR from the user's perspective.
-  * In section IV.E: "which is the trade-off the user actually cares about when reading the graph panel of the console." Given that there's no study of how users use the graph panel, especially with respect to different values of k, I don't think it's valid to claim to know what the user cares about.
-  * In section IV.F, I would appreciate some discussion of the suspected cause of the hazard handling degradation as training continues. It is not clear to me why that would happen.
-  * In section IV.F: "The system freezes at the best validation checkpoint," I'm a little confused here. Training occurs only once (the same model weights are used for different scenarios as mentioned in IV.A), so why is it that the //system// freezes? Shouldn't it be the //authors// who freeze training?
-  * In section IV.F, "About 5.8% of prompts produce parses that are syntactically valid but semantically off." I would appreciate some explanation of how this value was measured. It doesn't seem trivial to detect when the semantics of the JSON encoding differ from the prompt.
-===== Notes =====
-  * After a first reading, my main concern is that I don't know what the evaluation is supposed to prove about the implementation/design.
-    * The paper's evaluation is technical, but I would've expected an evaluation which centers on a users experience using the intermediate representation.
-    * This is especially considering that the ML technqiues don't seem to be a big contribution of this work, and they are somewhat minimized by the authors.
-    * I think a more effective evaluation strategy could be to propose more in-depth usage scenarios, as we saw in the example interaction of III.B.
-==== Approach ====
-  * The authors outline a 4-step transformation process which converts natural language into a CARLA simulation.
-    - An LLM generates a JSON object which encodes a traffic scenario, with one object per vehicle.
-    - The JSON representation is converted into a graph.
-      * **Question:** is this mapping unambiguous? It seems at first glance to be, since the LLM doesn't generate a graph, but a list of objects and their positions, from which the graph is induced.
-        * It turns out that it is ambiguous, since it can fail. And some detail about these failures is found in section IV.
-      * Each node contains parameters about the vehicle, such as its position or velocity.
-    - The graph representation is passed through a Graph Neural Network (GNN) which is meant to, at each timestep, predict a policy for each vehicle based on its relationship to the other vehicles.
-      * The weights of this network are trained from an existing simulation, HighwayEnv.
-        * **Question:** the authors claim that the learned network can attend to the role and style of the vehicle (such as aggressive, or cautious), but where does the source of truth for these behaviors come from? Does HighwayEnv already have a notion of cautious or aggressive?
-          * After looking at the GitHub page for HighwayEnv, I am still wanting for an answer.
-    - The policy decisions at each timestep are rendered using a CARLA simulation.
-  * The intermediate representation that the authors propose is a graph which is visually annotated with attention weights
-    * In that sense, the algorithm provided is important in determining and visualizing the attention.
-    * Can be seen as a kind of explainability, as proposed in GAT paper.
-  * But we don't get a sense of the interactions which are possible now that this intermediate representation exists.
-    * We see the user editing the velocity of the car, but this edit comes even before the graph and its attention mechanism are used.
-    * I would like to see what other interactions are possible //because// of this intermediate representation.
-      * For example, what if the wrong node is attended to? Can that happen? If so, how does the user correct it?
-      * What if the vehicle's behavior isn't as expected at the level of the Q-values? What recourse does the user have?
-      * What if the graph representation is not as the user expects? Can this be corrected?
-==== Evaluation ====
-  * Figure 4 demonstrates several prompts and renderings, but the interesting part here for me would've been the intermediate representations.
-  * I think the RL loop could be more clearly explained. My understanding is that HighwayEnv provides the ground truth behavior of the vehicles, and takes as input commands, the network producing those commands given a description of the graph of entities.
-  * In section IV.E, I'm not convinced of the explanation about why k!=5 would perform worse. For one, I think that "visual signal" should be further explained. Are you referring to the thicker line employed when two nodes attend to one another?
-    * Could it be that there is some bias in the training data towards graphs with degree 5 as opposed to 3 or 7?
-    * We don't know what the "user actually cares about" because there's no survey of users.
-  * In section IV.F, could you lend some intuition as to why you suspect the hazard handling degrades? It's not clear to me why that would happen.
-  * "The system freezes at the best validation checkpoint," I'm a little confused here. Training occurs only once, so why is it that the system freezes? Shouldn't it be the authors which freeze?
-  * "which is an observable property of the specification," which specification?
-  * "About 5.8% of prompts produce parses that are syntactically valid but semantically off" How did you measure this? It seems difficult to have a ground truth for semantics.
-==== Nits ====
-  * In section III.B, in paragraph 6, "the same interactions but with changed perceptual conditions have changed" -> perhaps you mean to remove "have changed."
-===== Rebuttal =====
-We thank the reviewers for the careful reading. Our contribution is a layered authoring surface that exposes every intermediate representation between sentence and simulation as an
-editable artefact. We respond below to specific questions and misunderstandings, organised by reviewer.
-Reviewer 1: R1's central concern is whether the user can meaningfully intervene when intermediate representations behave unexpectedly. The capabilities exist by design at each layer. Specifically:
-— Wrong node attended to in the graph: the graph panel exposes the k-NN connectivity parameters (k, d_max). Adjusting these causes the graph to re-render and the policy's attention to update accordingly. A salient vehicle excluded by the 100m threshold is brought into the attention budget by widening the radius.
-— Unexpected graph structure: node-edge composition follows from the JSON specification combined with connectivity parameters. The user intervenes at the JSON layer (vehicle role, lane, position) or at the parameter layer. Both edits are visible before any 3D render.
-— Unexpected simulation behaviour: the policy panel exposes the per-step attention heatmap and Q-values. The user can step forward and identify which stage produced the unexpected output whether its JSON, graph, or policy & correct at that layer rather than rewriting the sentence.
-We acknowledge that the paper illustrates these recourse paths only at the JSON layer in the current Section III-B walkthrough.
-R1 asks how HighwayEnv interprets "aggressive". HighwayEnv does not parse behavioural style strings. It consumes kinematic values (lane, position, velocity). The style-to-kinematics mapping lives in the parsing layer: the few-shot prompt template provided to Gemini contains worked examples that demonstrate what "aggressive" configurations look like in HighwayEnv terms (smaller front gap, higher relative velocity, lane-adjacent placement). Gemini learns the mapping from these examples and produces JSON with kinematic values consistent with the style label. The behavioural styles supported by the system are those illustrated in the few-shot examples. Styles outside that distribution may not produce kinematic outputs that align with the label.
-R1 questions whether the k≠5 performance drop reflects a fundamental property or training-distribution bias. He is correct that the policy was trained on k=5 graphs, so degradation at k=3 and k=7 cannot be cleanly disentangled from train/test distribution shift.
-We accept that "the trade-off the user actually cares about" overclaims in the absence of a user study. Regarding why hazard handling degrades as training continues, the
-mechanism is replay-buffer composition. Early in training, adversarial encounters dominate the replay buffer because the policy is unsuccessful at avoiding them. As the policy improves, near-collision states become rare in collected experience, and the policy increasingly optimises the velocity-reward term at the expense of collision-avoidance behaviour it no longer rehearses. We freeze at the best validation checkpoint (episode 500) for this reason.
-R1 asks how the 5.8% semantically off rate was measured. By automated schema validation comparing prompt tokens to populated JSON fields, a mismatch (e.g., prompt contains "truck", but the vehicle_type field is "car") is flagged. The check is conservative and the figure represents detected mismatches, not all possible semantic errors. The methodology is underspecified in IV-F.
-Other R1 points: (1.5) the title could foreground the intermediate representation more directly; (1.7) the full console screenshot should precede the detail view; (1.9) "visual signal" in IV-E refers to user-facing edge density in the graph panel when k is large; (1.10) showing graph representations alongside the HighwayEnv/CARLA frames in the scenario gallery would make the schema's generalisation visible at the user-facing layer; (1.13) "system freezes" is loose phrasing as we select the best validation checkpoint.
-Reviewer 2: R2 identifies the absence of a user study as the central weakness. The paper presents a working prototype and a technical characterisation of its behaviour and defends the accessibility claim by appeal to the working system and prior end-user-programming literature. R2 is correct that this is the natural next step for the work.
-R2 notes the comparison to Scenic, ScenarioRunner, Text2Scenario, LASER, ChatScene, and LCTGen is conceptual. This is accurate. The paper compares on the structural axis of intermediate representation visibility, where the comparison can be made from each system's public documentation. A task-based comparison on authoring effort, editability, transparency, or error-recovery workflows requires the controlled user study would be a natural next step for this line of work.
-R2 notes a tension between the gallery (highway, on ramp, roundabout, pedestrian, edge case) and the limitations statement that the corpus is mainly highway. The gallery illustrates the schema's expressive range across road geometries; the 2,500-prompt evaluation corpus underlying Section IV's numerical claims is restricted to highway scenarios. The gallery scenarios do not support the parsing-rate or executability numbers. The gallery caption should disambiguate this.
-R2 asks for more detail on corpus construction. The 2,500 prompts were generated by varying three worked example templates along three axes: paraphrase (lexical variation), vehicle count (3-10),
-and behavioural-style assignment (cautious/normal/aggressive). The corpus construction is undersupported in the present text.
-R3 questions whether AV practitioners need natural-language authoring given that engineers can write Scenic. We share the instinct, but the target audience is not the AV engineer who can write DSL code — it is the safety analyst, behavioural analyst, or domain expert whose expertise is in identifying which scenarios matter for safety validation, but who does not typically have programming background. This distinction could be sharper in the introduction.
-Reviewer 3: R3 raises LLM hallucination as a safety-critical concern. The schema-constrained parsing and the visible JSON intermediate representation catch the hallucinated content at the JSON layer before any safety-relevant artifact is produced. This is the purpose of making the intermediate representation user-facing rather than burying it behind a black-box pipeline.
-R3 raises concern that corpus prompts may be stylistically uniform. The paraphrase axis described above produces lexical variation across the 2,500 prompts; the 94.2% first-attempt parse rate is
-across this varied corpus, not across paraphrases of a single canonical prompt.
-R3 notes a GNN/GAT labelling inconsistency between pages 1 and 3. The architecture is a two-layer Graph Attention Network (GAT) throughout; the GNN label on page 1 is a typographic slip.