VLHCC 2017 Reviews

[jssmith1]

----------------------- REVIEW 1 ---------------------
PAPER: 54
TITLE: Flower: Navigating Program Flow in the IDE
AUTHORS: Justin Smith, Chris Brown and Emerson Murphy-Hill

Overall evaluation: 3 (strong accept)
Reviewer's confidence: 4 ((high))

----------- Review -----------
This short paper presents a program navigation tool. Program navigation is an area that is known to be problematic yet there is a paucity of exploratory research in the area. The ideas in flow-er are simple and seem to be effective for simpler navigation tasks. Of course, it is more complex navigation tasks that challenge developers and are more challenging to find solutions. Never-the-less this is a nice and timely contribution.

I particularly appreciate that the authors see the work as work-in-process and have appropriately submitted a short paper.

Thank you for your rebuttal comments. Good luck with the ongoing project.

----------------------- REVIEW 2 ---------------------
PAPER: 54
TITLE: Flower: Navigating Program Flow in the IDE
AUTHORS: Justin Smith, Chris Brown and Emerson Murphy-Hill

Overall evaluation: 2 (accept)
Reviewer's confidence: 4 ((high))

----------- Review -----------
The short paper proposes an extension to Eclipse that allows seamless navigation across uses of a variable from both control flow and data flow perspective. Navigation is integrated with the editing view, not as a separate window so doesn’t require context switching. A preliminary study suggests some efficacy with the approach but some deficiencies that need addressing.

This is a useful piece of preliminary work. The navigation integration is is a nice idea attempting to address issues with existing approaches. The study is a useful design aid that demonstrates areas for further improvement, but provides a conditional endorsement of the idea.

In summary, fits the brief of a short paper well.

----------------------- REVIEW 3 ---------------------
PAPER: 54
TITLE: Flower: Navigating Program Flow in the IDE
AUTHORS: Justin Smith, Chris Brown and Emerson Murphy-Hill

Overall evaluation: 1 (weak accept)
Reviewer's confidence: 4 ((high))

----------- Review -----------
The paper presents a new tool for control and data flow navigation within an IDE, together with a preliminary evaluation which yielded some observations on the utility. In general, the aim of exploring the minimal features required to achieve certain tasks, thereby facilitating the reduction of complexity of related interfaces is clearly worthwhile. Of course, if the overall desired task set is enlarged, things will become more complicated, but this seems a sensible approach to begin to develop understanding.

Regarding the evaluation, can you give more details about the data obtained #146, any hypothesis tested and the appropriateness of the tests applied #147 – e.g. relating to the variance of the samples? Has the preliminary evaluation enabled a clear hypothesis to test in follow up experimentation?

#150 In related work, phrases containing “generally differ” might be made more precise. #149 Is Flower the name of the tool or the approach or both?

Overall, the short paper appears to make a small contribution to knowledge, and may stimulate some interesting conversations at the conference.

---

Thanks for the response, which enhances belief that the update will improve the short paper further. Am happy to recommend accept as short paper.

----------------------- Rebuttal ---------------------
We thank the reviewers for their thoughtful feedback, which we look forward to addressing in revision.

#146 R3 asks about the data obtained. In addition to demographic information, we collected screen and audio recordings of each participant's tasks, including his/her training sessions. We analyzed the screen recordings to obtain task completion times and timestamps for relevant methods visited. Participants also provided PSSUQ scores for both tools. We recorded participants’ verbal responses to questions based on Nielsen’s usability heuristics. Lastly, we asked participants to report their confidence in each assessment, though these ratings are not discussed in the paper due to space limitations.

#147 R3 requests clarification about the statistical tests we applied. We tested whether participants performed differently — in terms of task completion time and correctness — using the full suite of Eclipse tools, compared with Flower. We first tested the distribution task completion times for normality using the Shapiro-Wilk test and failed to reject the null hypothesis (p=.21 and p=.43 for each task, respectively). Therefore, we tested for differences in task completion time by performing two-tailed, unpaired, two-sample t-tests. We tested for differences in task correctness using a chi square test, which is appropriate for nominal data.

#148 R3 asks about following up on this work. Our preliminary evaluation has encouraged us to continue exploring the idea of minimalistic program navigation tools and to continue making improvements to the Flower prototype (R1/R2). In follow-up experimentation, we also want to test whether other types of tools’ interfaces can similarly be made minimalistic.

#149 Finally, when we introduce Flower, we will clarify that it is the name of our prototype tool (R3).