research

Generative AI tools can help users with many tasks. One such task is data analysis, which is notoriously challenging for non-expert end-users due to its expertise requirements, and where AI holds much potential, such as finding relevant data sources, proposing analysis strategies, and writing analysis code. To understand how data analysis workflows can be assisted or impaired by generative AI, we conducted a study (n=15) using Bing Chat via participatory prompting. Participatory prompting is a recently developed methodology in which users and researchers reflect together on tasks through co-engagement with generative AI. In this paper we demonstrate the value of the participatory prompting method. We found that generative AI benefits the information foraging and sensemaking loops of data analysis in specific ways, but also introduces its own barriers and challenges, arising from the difficulties of query formulation, specifying context, and verifying results.

As search evolves and Generative AI enables users to express more complex information needs and goals, it is an opportune moment to investigate how the search for information influences creativity. Little is known about how creators — especially novices who lack domain-specific terminology — use web search when developing an idea, and vice versa, how new information shapes an idea. To investigate how ideas evolve through web search, we conducted an online lab study with 56 design students who engaged in a 3-week product redesign project. Through a mixed-method analysis of web search logs, surveys, and interviews, we report on the different search behaviors, strategies, challenges and four distinct patterns–Orienters, Refiners, Confirmers, and Pivoters–that illustrate how the impact of search depends on the maturity of an idea. We discuss design opportunities to enhance web search systems for ideation and pedagogical interventions to teach creators how to improve idea-centric search.

Advances in AI, particularly large language models (LLMs), can transform creative work. When developing a new idea, LLMs can help designers gather information, find competitors, and generate alternatives. However, LLM responses tend to be long-winded or contain inaccuracies, placing a burden on users to carefully synthesize information. In our formative studies with 52 students and five instructors, we find that novice designers typically lack guidance on how to compose prompts, reflect critically on LLM responses, and extract key information to help shape an idea. Building on these insights, we explore an alternative approach for interacting with LLMs, not via chat, but rather through structured templates. Collaborative design templates are a well-established strategy for helping novices think, organize information, and reflect on creative work. Developed as a digital whiteboard plugin, Jamplate integrates LLM capabilities into design templates, streamlining the collection and organization of user-generated content and LLM responses within the template structure. In a preliminary study with 8 novice designers, participants expressed that Jamplate’s reflective questions and in-situ guidance improved their ability to think critically and improve ideas more effectively. We discuss the potential of designing LLM-enhanced templates to instigate critical reflection.

Creating truly original ideas requires extensive knowledge of existing ideas. Navigating prior examples can help people to understand what has already been done and to assess the quality of their own ideas through comparison. The creativity literature has suggested that the conceptual distance between a proposed solution and a potential inspiration can influence one's thinking. However, less is known about how creators might use data about conceptual distance when exploring a large repository of ideas. To investigate this, we created a novel tool for exploring examples called IdeateRelate that visualizes 600+ COVID-related ideas, organized by their similarity to a new idea. In an experiment that compared the IdeateRelate visualization to a simple list of examples, we found that users in the Viz condition leveraged both semantic and categorical similarity, curated a more similar set of examples, and adopted more language from examples into their iterated ideas (without negatively affecting the overall novelty). We discuss implications for creating adaptive interfaces that provide creative inspiration in response to designers' ideas throughout an iterative design process.

While most crowd work seeks consistent answers, creative domains often seek more diverse input. The typical crowd mechanisms for controlling quality may stifle creativity, yet removing them altogether could just produce noise. Schemas and metadata provide two mechanisms for embedding existing knowledge into task environments. Schemas are expert-derived patterns designed to structure how people think through a problem. Metadata, on the other hand, illustrate a range of creative input that fits within the structure of a schema. To understand the relative effects of schemas and metadata, we conducted a study where crowd workers are asked to generate creative interpretations for a set of placemaking examples. Crowd workers were guided either by schema plus metadata, schema alone, or neither. We found that showing schema along with crowd-produced metadata helped workers contribute interpretations that are both more on-topic and diverse, compared to using the schema alone or no schema. We discuss the implications on how crowds can creatively build on insights shared by others.

From Fortune 500 companies to local communities, organizations often strive to build a shared understanding about complex problems. Design competitions provide a compelling approach to create incentives and infrastructure for gathering insights about a problem-space. In this paper, we present an analysis of a two-month civic design competition focused on transportation challenges in a major US city. We examine how the event structure, discussion platform, and participant interactions affected how a community collectively discussed design constraints and proposals. Ninety-two participants took part in the competition's online discussion, hosted on Slack. Applying a mixed-methods analysis, we found that participants shared less as they settled into teams and, due to the discussion system, had difficulty seeing how topics connected across channels; we also learned that certain messages led participants to add depth to existing topics. Based on the findings we provide recommendations for civic competitions aimed at building knowledge around a problem.

This paper describes the use of the Delphi method as a means of incorporating study participants into the processes of data analysis and interpretation. As a case study, it focuses on perceptions about use and non-use of the social media site Facebook. The work presented here involves three phases. First, a large survey included both a demographically representative sample and a convenience sample. Second, a smaller follow-up survey presented results from that survey back to survey respondents. Third, a series of qualitative member checking interviews with additional survey respondents served to validate the findings of the follow-up survey. This paper demonstrates the utility of Delphi by highlighting the ways that it enables us to synthesize across these three study phases, advancing understanding of perceptions about social media use and non-use. The paper concludes by discussing the broader applicability of the Delphi method across CSCW research.