Dear Authors, I've emailed 2 reviewers and encouraged them to reply to your responses.

Best,

PC

I've submitted the IRB determination on the requested Google Form. Please note that our IRB deemed our study exempt, despite mentioning that the game is rated M for mature. This is likely because we only include participants who have previously played the game and are at least 18 years of age without collecting demographic information about the users.

Thank you for the response and the additional evidence!

Under reasons to accept, you describe our work as providing valuable insights. Thanks for noticing! We appreciate your feedback and sincerely wish that you'll read our comments and reflect on them during the reviewer discussion period. If you do, we hope that you will update your score and associated review text to indicate how our comments below factor into your decisions. Thanks!

Background

We wish to bring your attention to an important consideration for our work: while many companies are looking to integrate LLMs into video games, no published research has performed an extrinsic evaluation of model-generated dialogue in a video game, not even from major game studios like Microsoft [1] who have direct access to the game code. The referenced paper (which we only became aware of after submission and will cite in a revision of the paper) explores dialogue generation for the roleplaying video game The Outer Worlds, but in contrast only looks at utterances without conditioning on game state and uses a mix of automatic metrics and out-of-context human evaluation. In comparison, we reverse engineer the dialogue system from Disco Elysium and recreate it in a web app environment which reduces confounding variables (e.g. choosing where to go, who to interact with, etc). The release of our code and tools will democratize the ability to conduct future in-context human evaluation for research exploring interactive dialogue in video games. This is important since LLMs are actively being investigated for video games and other virtual experiences as mentioned in our paper: many well-funded startups like InWorld AI [2], character.ai [3], and convai [4] have begun offering products in this space, and established gaming companies like Microsoft [1] and Ubisoft [5] are conducting research in the area too.

Study Size

Your first stated concern is whether our usage of the term "large-scale" is appropriate. Underlying that concern is an important question: Are 28 players enough to provide meaningful insights into our research question regarding player experience with LLM-generated dialogue? We begin by noting that our paper ensures that 100 utterances receive at least three annotations across our 28 participants as discussed in our Study parameters in Section 4.3. Using the methodology from [6], our study design has a statistical power of 0.96 for a margin of 10% in a high variance population (see the tables at the end for a more complete analysis). The observed margins in our study as described in Section 5.1 are often much larger (e.g. in aggregate 61% of players prefer human-written dialogue, a margin of 22%) indicating more than sufficient statistical power for our reported results (a power $\ge 0.8$ is typically desired). The additional fine grained analysis in Section 5.2 covers specific actionable areas of concern to improve modeling for the task in the future. Therefore we believe the scale of the study is adequate for the findings we report. We will include a discussion of our study's statistical power in a revision of the paper and will happily address your concern by not referring to our study as "large-scale".

Dynamic Text Generation

Your second concern is whether our evaluation is dynamic enough. We discuss why we make this choice in our limitations section. Including a player's dialogue history as part of the prompt is an easy change to include in our framework, but we chose not to do so as it may introduce a large amount of variance necessitating a considerably larger number of participants to achieve a similar statistical power as our current design. This is certainly a task that can be explored in future research once our code is released. Though please note that our work goes above and beyond any other research in the field as discussed above [1].

Question 1: Terminology

A dialogue fork is a node with an outgoing edge where traversal is gated by a binary game state expression -- the edge can only be traversed if the game state matches a given Lua condition. We will clarify such terms further in a revision of the paper.

Question 2: Undefined Initials

You are correct, H is for human and G is for GPT-4 in those paragraphs. We will clarify this in a revision of the paper.

Question 3: Dialogue History

We are not sure we fully understand the question, but will try our best to answer. The prompts we use for dialogue generation do not contain any dialogue history, rather they include the game writer's comments and the alternate dialogue options conditioned upon game state. For an example, please see the Lua script in Step 2 of Figure 1. In regards to player annotation, we allow players to edit their feedback for a particular utterance after seeing future utterances.

Thanks for reading!

In light of our comments addressing your concerns, we sincerely hope you'll consider raising both your soundness and excitement scores. Specifically, we believe we provide sufficient support for all of our claims and that our work will enable a promising new research direction for evaluating video game dialogue in an interactive setting, which is an area of growing interest [1, 2, 3, 4, 5].

[1]: https://arxiv.org/abs/2212.10618
[2]: https://inworld.ai/blog/inworld-valued-at-500-million
[3]: https://www.crunchbase.com/organization/character-ai
[4]: https://investor.nvidia.com/news/press-release-details/2023/NVIDIA-ACE-for-Games-Sparks-Life-Into-Virtual-CharactersWith-Generative-AI/default.aspx
[5]: https://aclanthology.org/2023.nlp4convai-1.11/
[6]: https://aclanthology.org/2020.emnlp-main.745/

We report the results of the statistical power analysis of our study design below:

Num Workers: 10
Margin | Power | Variance
0.05   | 0.765 | low
0.05   | 0.325 | high
0.1    | 1     | low
0.1    | 0.775 | high
0.2    | 1     | low
0.2    | 1     | high

Num Workers: 28
Margin | Power | Variance
0.05   | 0.91  | low
0.05   | 0.475 | high
0.1    | 1     | low
0.1    | 0.96  | high
0.2    | 1     | low
0.2    | 1     | high

Num Workers: 50
Margin | Power | Variance
0.05   | 0.925 | low
0.05   | 0.61  | high
0.1    | 1     | low
0.1    | 1     | high
0.2    | 1     | low
0.2    | 1     | high

Thank you for acknowledging our rebuttal. We don't see any updates to the text of the review, though we are happy to see you increased your excitement score.

Could you please let us know your thoughts on the soundness of our work. What concerns do you still have that are not addressed in our rebuttal? Which claims do you think need extra support? Thanks for your time!

Thank you for your thoughtful comments and the updates to your review after our rebuttal. We will make sure to include your suggested improvements in the next revision of our paper.

Under reasons to accept, you state that our dataset, task, and experiment design are interesting. Thanks for the kind words! We appreciate your feedback and sincerely wish that you'll read our comments and reflect on them during the reviewer discussion period. If you do, we hope that you will update your score and associated review text to indicate how our comments below factor into your decisions.

Background

We wish to bring your attention to an important consideration for our work: while many companies are looking to integrate LLMs into video games, no published research has performed an extrinsic evaluation of model-generated dialogue in a video game, not even from major game studios like Microsoft [1] who have direct access to the game code. The referenced paper (which we only became aware of after submission and will cite in a revision of the paper) explores dialogue generation for the roleplaying video game The Outer Worlds , but in contrast only looks at utterances without conditioning on game state and uses a mix of automatic metrics and out-of-context human evaluation . In comparison, we reverse engineer the dialogue system from Disco Elysium and recreate it in a web app environment which reduces confounding variables (e.g. choosing where to go, who to interact with, etc). The release of our code and tools will democratize the ability to conduct future in-context human evaluation for research exploring interactive dialogue in video games. This is important since LLMs are actively being investigated for video games and other virtual experiences as mentioned in our paper: many well-funded startups like InWorld AI [2], character.ai [3], and convai [4] have begun offering products in this space, and established gaming companies like Microsoft [1] and Ubisoft [5] are conducting research in the area too.

Modeling Concerns

We want to first address your concern regarding our focus on evaluation over modeling for this task. We greatly appreciate that you notice the considerable effort we spent on our evaluation! We believe this aligns very closely with the theme of the Human-Centered NLP track our paper was submitted to. When this track was first announced as a special track at NAACL 2022 [6], it was made clear that human factors are the focus rather than modeling contributions (emphasis added):

If the submission contributes a new modeling or training technique that is evaluated on standard corpus-based benchmarks, it would best fit the regular track. However, if the submission evaluates, e.g., how users calibrate their trust in the predictions of the system, it would be a good fit for the theme track.

Our player-centered analysis in Section 5 clearly aligns with the stated goal above.

That said, while we do not focus heavily on the modeling for this task, we experiment with both fine-tuning and few-shot approaches as described in Appendix C. We use Lua to ground the dialogue with game state as the Disco Elysium dialogue system is Lua-based and there is no way to directly translate Lua code into natural language. Therefore we focus on models which have shown strong performance on both code and language domains like GPT-4. Additionally, by formalizing the dialogue structure as Lua we open the possibility to conduct future research that allows for dynamically changing the game state based on machine-generated utterances, an open problem.

Linguistic Analysis

As you can see from Section 5.2, we include a fine-grained analysis performed by players comparing human-written and machine-generated dialogue. This analysis includes a discussion of observed linguistic features, such as awkward articulations (e.g. combining the use of first person and third person pronouns within a single utterance) and interjecting spurious phrases during paraphrasing.

Style Identification

If you look at the results highlighted in Figure 3b and discussed in Section 5.1, players are not merely performing style identification. The most common reason for changing their preference from LLM-generated to human-written dialogue is that the utterance is illogical in context. Though since our paper is concerned with human-centered factors, even if players do perform style identification, that would still be an important consideration: one of the largest video game publishers, Ubisoft, is actively researching style for video game dialogue to ensure consistency [5].

Ethics

In regards to your ethical concerns, we take great care to respect the copyright of the work. We want to make it clear: we do not directly release a dataset . Rather, we release instructions for using AssetStudio [7] to extract the game data from a legally obtained copy of the game . We then provide tools for processing this data allowing easier replication and enabling future experimentation by researchers. Finally, our study requires a login with a registered account only available to participants who have previously completed the game.

Question A: Take-home Message

As we highlighted in the Background above, LLMs are actively being investigated for video games. Despite this interest, there is no published research on player experience interacting with the LLMs in a game setting. Our research provides both macro- and fine-grained analysis of specific issues LLMs face in this context.

Question B: Familiarity

As described in Section 4.3, players need to be experienced with the game because:

• Dialogue choices in Disco Elyisum affect the game state and change what dialogue options the player has available in the future. This is an acquired skill we ensure participants have by limiting participation to those who have played the game.
• The conversation players are evaluating are from our validation set (the dataset split was automatically determined by minimizing game state overlap as described in Section 2.1). This conversation is from late in the game and thus requires familiarity with the game to understand and assess the conversation, e.g. we wouldn't expect someone to understand references in the last episode of a TV show if they haven't watched the rest of the show; the same consideration applies here.

Please note that from our pilot and observational study it became clear that players could not remember exact dialogue options. As the game was originally released in 2019 with an updated release in 2021, most players had completed their play-through significantly earlier than our main study's timeframe in Spring of 2023.

Question C: Study Cost

We had 5 participants for our pilot study, 2 for our observational study, and 28 for our main study. In total, we spent nearly 1000 dollars in gift cards across all participants (a breakdown of gift card amounts is detailed in Section 9). We further spent approximately 3000 dollars in cloud costs for both our fine-tuning and few-shot experiments. Finally, we spent 1.5 years of research time to produce this study, starting from Jan 2022 -- an exact salary breakdown is difficult to provide.

Thanks for reading!

In light of our comments addressing your concerns, we sincerely hope you'll consider raising both your soundness and excitement scores. Specifically, we believe we provide sufficient support for all of our claims and that our work will enable a promising new research direction for evaluating video game dialogue in an interactive setting, which is an area of growing interest [1, 2, 3, 4, 5].

[1]: https://arxiv.org/abs/2212.10618
[2]: https://inworld.ai/blog/inworld-valued-at-500-million
[3]: https://www.crunchbase.com/organization/character-ai
[4]: https://investor.nvidia.com/news/press-release-details/2023/NVIDIA-ACE-for-Games-Sparks-Life-Into-Virtual-CharactersWith-Generative-AI/default.aspx
[5]: https://aclanthology.org/2023.nlp4convai-1.11/
[6]: https://2022.naacl.org/blog/special-theme/
[7]: https://github.com/Perfare/AssetStudio

Review Problem Type: G. The review does not match the type of contribution (e.g. experimental work expected of a paper stating a different kind of contribution)

Please try to get the reviewers to actually read our rebuttal and discuss it before updating their scores. We submitted our paper to the Human-Centered NLP track because it closely aligns with our research focus. In fact when the special theme of Human-Centered NLP was announced at NAACL 2022, our work was cited as an example of the types of papers that fit the theme. While this reviewer correctly recognizes the considerable effort we make for our human evaluation, they cite a lack of modeling contributions as a weakness of our work, a criticism we've unfortunately heard many times in the past.

Thanks in advance!

Thank you for acknowledging our rebuttal. We don't see any updates to the text of the review, though we are happy to see you increased your excitement score.

Could you please let us know your thoughts on the soundness of our work. What concerns do you still have that are not addressed in our rebuttal? Which claims do you think need extra support? Thanks for your time!

Under reasons to accept, you describe our work as interesting and meaningful. We're thankful you noticed! We appreciate your feedback and sincerely wish that you'll read our comments and reflect on them during the reviewer discussion period. If you do, we hope that you will update your score and associated review text to indicate how our comments below factor into your decisions.

Background

We wish to bring your attention to an important consideration for our work: while many companies are looking to integrate LLMs into video games, no published research has performed an extrinsic evaluation of model-generated dialogue in a video game, not even from major game studios like Microsoft [1] who have direct access to the game code. The referenced paper (which we only became aware of after submission and will cite in a revision of the paper) explores dialogue generation for the roleplaying video game The Outer Worlds , but in contrast only looks at utterances without conditioning on game state and uses a mix of automatic metrics and out-of-context human evaluation . In comparison, we reverse engineer the dialogue system from Disco Elysium and recreate it in a web app environment which reduces confounding variables (e.g. choosing where to go, who to interact with, etc). The release of our code and tools will democratize the ability to conduct future in-context human evaluation for research exploring interactive dialogue in video games. This is important since LLMs are actively being investigated for video games and other virtual experiences as mentioned in our paper: many well-funded startups like InWorld AI [2], character.ai [3], and convai [4] have begun offering products in this space, and established gaming companies like Microsoft [1] and Ubisoft [5] are conducting research in the area too.

Generalizability

While we do mention that our work does not generalize to all narrative-driven games, the framework we put forth is indeed useful across many games: any video game that provides dialogue choices conditioned on game state. These types of games are played by millions of players: see Baldur's Gate 3 [6] as a very recent example of their popularity (see also the examples we highlight in our Background above). Note that Disco Elysium uses an off-the-shelf dialogue system made for the Unity game engine which is in use by many additional games [7, 8]. This means the tools we release can be directly used to to explore and conduct experiments on a wide range of currently released video games.

Study Size and Heterogeneous Raters

We appreciate the concern over the study size in the face of potentially heterogeneous raters. As you note, our paper describes a study where 100 utterances receive at least three annotations across our 28 participants as discussed in our Study parameters in Section 4.3. Using the methodology from [9], our study design has a statistical power of 0.96 for a margin of 10% in a high variance population (see the tables at the end for a more complete analysis). The observed margins in our study as described in Section 5.1 are often much larger (e.g. in aggregate 61% of players prefer human-written dialogue, a margin of 22%) indicating more than sufficient statistical power for our reported results (a power $\ge 0.8$ is typically desired). The additional fine grained analysis in Section 5.2 covers specific actionable areas of concern to improve modeling for the task in the future. Therefore we believe the scale of the study is adequate for the findings we report even across differences such as play style. We will include a discussion of our study's statistical power in a revision of the paper.

Familiarity Bias

As described in Section 4.3, players need to be experienced with the game because:

• Dialogue choices in Disco Elyisum affect the game state and change what dialogue options the player has available in the future. This is an acquired skill we ensure participants have by limiting participation to those who have played the game.
• The conversation players are evaluating are from our validation set (the dataset split was automatically determined by minimizing game state overlap as described in Section 2.1). This conversation is from late in the game and thus requires familiarity with the game to understand and assess the conversation, e.g. we wouldn't expect someone to understand references in the last episode of a TV show if they haven't watched the rest of the show; the same consideration applies here.

Please note that from our pilot and observational study it became clear that players could not remember exact dialogue options. As the game was originally released in 2019 with an updated release in 2021, most players had completed their play-through significantly earlier than our main study's timeframe in Spring of 2023.

Additional References

We would be happy to include a discussion of additional references and how they relate to our work.

Question A: Additional Instructions

In addition to the instructions seen in Figure 2 of our paper, raters were given the following instructions:

As you play, you will be presented with dialogue options that may be machine-generated, paired alongside the original dialogue from the game (e.g. 1a vs 1b ). Anytime you choose a paired dialogue option you will be prompted to state why you chose the option. You will then be presented the next line of dialogue that will be spoken and asked to reassess your choice. Please consider the next line of dialogue and the historical context of the dialogue thus far when justifying your choices (e.g., because your choice fit the context better than the other choice or was more interesting).

• Please play through the scenario as many times as needed to exhaust all the dialogue options.
• We highly encourage you to include quotations from the dialogue history or the choices in your justifications!

Question B: Why Lua?

We use Lua to ground the dialogue with game state as the Disco Elysium dialogue system is Lua-based and there is no way to directly translate Lua code into natural language. Therefore we focus on models which have shown strong performance on both code and language domains like GPT-4. Additionally, by formalizing the dialogue structure as Lua we open the possibility to conduct future research that allows for dynamically changing the game state based on machine-generated utterances, an open problem.

Question C: Demographics

We specifically do not collect demographic information in order to conform to the requirements of our Institutional Review Board (IRB), only ensuring that participants are at least 18 years of age and have completed a play-through of the game.

Question D: Study Generalizability

Please refer to our discussion of Study Size and Heterogeneous Raters above.

Thanks for reading!

In light of our comments addressing your concerns, we sincerely hope you'll consider raising both your soundness and excitement scores. Specifically, we believe we provide sufficient support for all of our claims and that our work will enable a promising new research direction for evaluating video game dialogue in an interactive setting, which is an area of growing interest [1, 2, 3, 4, 5].

[1]: https://arxiv.org/abs/2212.10618
[2]: https://inworld.ai/blog/inworld-valued-at-500-million
[3]: https://www.crunchbase.com/organization/character-ai
[4]: https://investor.nvidia.com/news/press-release-details/2023/NVIDIA-ACE-for-Games-Sparks-Life-Into-Virtual-CharactersWith-Generative-AI/default.aspx
[5]: https://aclanthology.org/2023.nlp4convai-1.11/
[6]: https://wccftech.com/baldurs-gate-3-topped-5-2-million-units-sold-on-steam-says-belgian-embassy/
[7]: https://www.pixelcrushers.com/games-showcase/
[8]: https://www.articy.com/en/showcases/
[9]: https://aclanthology.org/2020.emnlp-main.745/

We report the results of the statistical power analysis of our study design below:

Num Workers: 10
Margin | Power | Variance
0.05   | 0.765 | low
0.05   | 0.325 | high
0.1    | 1     | low
0.1    | 0.775 | high
0.2    | 1     | low
0.2    | 1     | high

Num Workers: 28
Margin | Power | Variance
0.05   | 0.91  | low
0.05   | 0.475 | high
0.1    | 1     | low
0.1    | 0.96  | high
0.2    | 1     | low
0.2    | 1     | high

Num Workers: 50
Margin | Power | Variance
0.05   | 0.925 | low
0.05   | 0.61  | high
0.1    | 1     | low
0.1    | 1     | high
0.2    | 1     | low
0.2    | 1     | high