Venue: VL/HCC '26
Authors: Anonymous

The authors present Diffmagic, a code-diff visualizer which uses name resolution to organize the diff into separable chunks. They evaluate Diffmagic through a user study which measures the participants' performance in completing two tasks.

Thank you to the authors for submitting this paper. I think it would make an excellent contribution to VL/HCC, and I enjoyed reading it. In particular, I think that the related work is well explained in the context of this paper's contribution, and the user study, though limited in sample, provides good evidence that Diffmagic is usable and useful.

I recommend that this paper be accepted, but I do have some minor feedback which I have arranged by section. All of these points are minor, and I don't consider them necessary conditions for acceptance.

• In the introduction, Diffmagic “score[s] each change in a batch using reachability via transitive name-resolution edges.” Since we don't yet know the structure over which name resolution is being represented, I think the word “transitive” doesn't yet have meaning. It seems like a less precise explanation might suffice here.
• Section III is I think the first time that Figure 1 is referenced. It would be convenient for the reader if this figure were placed at the top of the same page.
• In Figure 3, I am confused what the purple edges represent. In the caption, they are described as name resolution edges, but based on the description in the 7th paragraph of IV.A, I understand the edges connecting the xs and ys to be supplemental parallel edges. It might be helpful if the caption included broader description of the purple edges, or differentiated the two kinds of (purple) edges.
• In IV.C, “Computing Causality Scores”, I am struggling to think of a situation where the causality graph would have any paths of length > 1. This seems to be true of the examples of parallel edges given in IV.A, and it is true of Figure 3, once the unchanged AST nodes are thrown out. It also follows my intuition that any name in binding position has only inbound edges, and any name in use position has only outbound edges. Do causality chains arise from the merging of rhs and lhs graphs?
• This question arises only because I'm trying to justify the complexity of the procedure for computing causality scores.
• It may be better for Figure 5 to appear near Figure 4, and the top half of Figure 4 to appear where Figure 5 is, to better match the text.
• In “Using batches to validate hypothesis,” there is a typo. “except P14 dit” → “except P14 did”

• The approach:
  1. For each variable/definition which has been added by the diff, find its binding/use.
  2. Extract a graph connecting each variable to its use, so long as both the variable and the use are “novel” (i.e. part of the diff), for both the before and after programs.
  3. Find the connected components of the before and after graphs.
  4. Try to match up the components from the before/after using a heuristic which looks for unchanged parts of the AST to connect to.

• The related work section is thorough, and for each piece of related work, describes how it relates to the paper.
• The idea itself is pretty clever, and I could see how it would make a significant improvement in understanding diffs.
• The user study seems well considered.
  • Though the sample is small, I take the likert scale portion of the study with more weight since it is a between-subjects study.
  • The likert scale suggests that users find diffs easier to reason about when using Diffmagic (Q4: “the order in which changes presented was logical”),
  • but that Diffmagic may not help much with very large diffs, since users agreed that (Q2) “the size of the diff impacted the difficulty of the task.”

• The main weakness of the paper in my view is the description of the algorithm, which I think could be a little further refined.
  • One way would be to extend Fig 3. with a subfigure showing the causality graphs.
  • I also think it would be more informative to see the induced parallel edges rather than the name resolution edges.

We thank the reviewers for their valuable comments and suggestions. Anything we don’t directly address below, we agree with and will address in the revision.

An overview of our intended changelist: * Include an informal explanation of “transitive name-resolution edges” * Reference Figure 1 in the introduction * Use different color for supplemental edges in Figure 3 * Add the effect size data from the answer below * Additional limitations and future work mentioned by reviewers

We now address individual questions and comments.

R1: I am struggling to think of a situation where the causality graph would have any paths of length > 1

Paths of length > 1 are possible, and as observed by the reviewer, they are the result of parallel edges, mainly inheritance-related ones. For example, assume class B inherits method f from class A, then some code uses f from an object of type B. There is a name resolution edge from the call site (use location) to B’s definition of f (binding location). However the path we would need to track renames is from f’s name in the call site to f’s name in B’s definition, and from the name f in B’s definition of f to the name f in A’s definition of f (both parallel edges). The same paths exist for parameters and arguments and for return types.

R1: do causality chains arise from the merging of rhs and lhs graphs?

No, when the grouped changes from the lhs and rhs are merged, no edge is added between them; the batch is conceptual.

R2: given the small sample size, effect sizes (e.g., Cohen's d) might be more informative than significance testing.

The following are the Cohen’s d results for time and comprehension in our tasks: Task 1 time: 0.06, Task 2 time: 1.40 Task 1 comprehension: 0.24, Task 2 comprehension: 0.23

We are happy to include them in the paper alongside significance testing.

R3: the compiling commit assumption

In general, Diffmagic is mainly intended for use in projects/organizations that use “compiling commit” as a policy; other diffs are not necessarily in its scope. Our participants, experienced industry developers, agreed that this is a reasonable assumption.

A class of “non-compiling” commits that Diffmagic already supports is one where not all project files are in the diff: Diffmagic only looks at changed files, a subset that will usually not compile on its own. This means it is already equipped to deal with code that does not fully compile. This is described in the paper and we are happy to further emphasize this.

Even when the commit is broken for additional reasons, Diffmagic is still “best effort”. This means it can generate any batches that correspond to the bindings that are resolved by its name resolution (this is mentioned in Section VII.C). For example, we have found that even if the code does not parse properly, any names that are found around the parser's error recovery may still resolve, and Diffmagic will batch the information that is available. In our experience for commits that are only a little broken, Diffmagic still does well, creating most of the batches. Diffmagic is not intended for commits that are really broken.

R3: mechanism to organize or rank the top-level batches themselves

We agree that organizing the batches is an important problem, and we note it in Section VII.C as a limitation. We do have some trivial ideas for solving this, e.g., sorting the batches by size, number of roots or total incoming degree of the roots, etc. Other more complex ideas, like exploring the effects of an LLM on this problem, would require new design work. This is an entire problem that requires its own design, methodology and algorithm and we feel that it is beyond the scope of this paper. We’re eager to address it in future work and in the meantime we will add some of these directions as future work next to the limitation in Section VII.C.

R3: Comparison to LLMs

Section VII.B discusses how LLMs can be used for tools like Diffmagic. As mentioned, LLMs can indeed be used for diff comprehension especially as they continue to improve. Even within Diffmagic, there are places where LLMs can be leveraged, for example, naming and organizing batches. To R3’s point about whether the nondeterministic nature of LLMs matters, a code diff is often looked at by developers multiple times (and even from multiple computers) and having a consistent view of the diff is useful for navigation. This indeed aligns with what we heard from one of our more experienced participants: P14 was initially worried that Diffmagic was entirely LLM-driven but was glad to learn about its more principled compiler-based approach.

R2, R3: Additional user study experiments and results

R2 and R3 had suggestions for additional experiments and results (e.g., additional types of tasks, comparison to LLM-driven summarization, have more scorers so we can report scoring reliability and IRR) and we agree those would have strengthened our results, and would be interesting to explore in follow-up work. We will gladly mention them in the discussion section under limitations and future work (section VII.C).