IlluminAItion
Topics:
Generative AI
IlluminAItion is a web game designed to introduce tech-savvy users to the concept of guardrails, or content filters, and demonstrate their role in mitigating biases in generative AI (GenAI). This project was part of the User-Centered Research and Evaluation (UCRE) class at CMU. Midway through the project, I played a key role in navigating the transition from a four-person to a three-person team, ensuring the project stayed on track despite challenges. My attention to detail and systems thinking helped maintain alignment with our goals and advanced the insights we gained through various research methods.
Roles:
Researcher, designer
Duration:
Aug 2024 - Dec 2024
Tools:
Figma
IlluminAItion
Topics:
Generative AI
IlluminAItion is a web game designed to introduce tech-savvy users to the concept of guardrails, or content filters, and demonstrate their role in mitigating biases in generative AI (GenAI). This project was part of the User-Centered Research and Evaluation (UCRE) class at CMU. Midway through the project, I played a key role in navigating the transition from a four-person to a three-person team, ensuring the project stayed on track despite challenges. My attention to detail and systems thinking helped maintain alignment with our goals and advanced the insights we gained through various research methods.
Roles:
Researcher, designer
Duration:
Aug 2024 - Dec 2024
Tools:
Figma
The Problem
The Problem
Generative AI (GenAI) tools are riddled with biases, but people do not have the tools nor the knowledge to identify and report these biases.
Generative AI (GenAI) tools are riddled with biases, but people do not have the tools nor the knowledge to identify and report these biases.
Current Solutions
Current Solutions
Existing GenAI auditing tools are confusing, have overly complex navigation, and contain numerous bugs that hinder usability.
Existing GenAI auditing tools are confusing, have overly complex navigation, and contain numerous bugs that hinder usability.
We evaluated WeAudit, a tool created at CMU, using Nielsen’s heuristics and think-aloud usability testing with four participants. Affinity clustering revealed key issues: unclear purpose, an inaccurate image model, and a confusing forum interface. These problems caused user confusion, limited engagement, and underutilization. Based on our insights, we concluded that a successful tool needs to have up-to-date technology, clear wording with bolded purpose statements, and well-structured information architecture. Applying to our initial target audience of people with limited knowledge of generative AI bias, we wanted to prioritize meaningful purpose, clear results, and ease of use.
We evaluated WeAudit, a tool created at CMU, using Nielsen’s heuristics and think-aloud usability testing with four participants. Affinity clustering revealed key issues: unclear purpose, an inaccurate image model, and a confusing forum interface. These problems caused user confusion, limited engagement, and underutilization. Based on our insights, we concluded that a successful tool needs to have up-to-date technology, clear wording with bolded purpose statements, and well-structured information architecture. Applying to our initial target audience of people with limited knowledge of generative AI bias, we wanted to prioritize meaningful purpose, clear results, and ease of use.
Reframing and Narrowing Our Scope
Reframing and Narrowing Our Scope
Using the “walk the wall” method to synthesize our initial research, we found that users are unmotivated to overcome the learning curves of existing tools, limiting their effectiveness. We then brainstormed design ideas focused on onboarding and motivation. After that, we explored the worst possible ideas, which led us to prioritize identification of biases in GenAI results. Usability testing further validated this, as users were unclear about what constitutes algorithmic biases. From these insights, we developed the question: How can we increase awareness of algorithmic biases among people with limited knowledge of GenAI, so they feel confident in identifying them?
Using the “walk the wall” method to synthesize our initial research, we found that users are unmotivated to overcome the learning curves of existing tools, limiting their effectiveness. We then brainstormed design ideas focused on onboarding and motivation. After that, we explored the worst possible ideas, which led us to prioritize identification of biases in GenAI results. Usability testing further validated this, as users were unclear about what constitutes algorithmic biases. From these insights, we developed the question: How can we increase awareness of algorithmic biases among people with limited knowledge of GenAI, so they feel confident in identifying them?
While we initially focused on bias reporting, our synthesis processes led us to the underlying issue of bias identification.
While we initially focused on bias reporting, our synthesis processes led us to the underlying issue of bias identification.
Contextual Research via Directed Storytelling
Contextual Research via Directed Storytelling
After receiving feedback from our TA, we narrowed our focus to ML developers, as they can address biases more directly. Using directed storytelling, we interviewed four AI/ML college students to explore the barriers and motivations they face. From these interviews, we gained key insights: developers are confident in identifying biases but struggle to understand the mechanisms behind them, highlighting the need for more transparency and education. Participants also noted that AI companies often treat bias as a public relations issue rather than a technical one. Finally, we realized that our inability to spot AI biases reflects our own blind spots—if we can't recognize bias, we can't expect machines to.
After receiving feedback from our TA, we narrowed our focus to ML developers, as they can address biases more directly. Using directed storytelling, we interviewed four AI/ML college students to explore the barriers and motivations they face. From these interviews, we gained key insights: developers are confident in identifying biases but struggle to understand the mechanisms behind them, highlighting the need for more transparency and education. Participants also noted that AI companies often treat bias as a public relations issue rather than a technical one. Finally, we realized that our inability to spot AI biases reflects our own blind spots—if we can't recognize bias, we can't expect machines to.
There needs to be greater transparency in AI systems, and tech-savvy people need to be equipped to prevent biases at the source.
There needs to be greater transparency in AI systems, and tech-savvy people need to be equipped to prevent biases at the source.
Aside on Navigating Conflict
Aside on Navigating Conflict
Conflict within our team began during the Worst Possible Idea activity when two members struggled to collaborate. Tensions escalated, and the third team member and I attempted to mediate, eventually involving the professor. Prior to the incident, one of the involved members had already shown signs of differing priorities and an unwillingness to compromise. Despite a rose-bud-thorn activity with the professor, the tension persisted. During this period, the team member repeatedly was absent from group meetings and did not contribute to the work, so I took on a larger share of the work to keep the project on track. Along with the other mediator, I led multiple conversations attempting to collaborate effectively and resolve tension. Ultimately, that person left the team, and the remaining three of us continued, with me taking the lead to ensure our work aligned with our overarching goals.
Conflict within our team began during the Worst Possible Idea activity when two members struggled to collaborate. Tensions escalated, and the third team member and I attempted to mediate, eventually involving the professor. Prior to the incident, one of the involved members had already shown signs of differing priorities and an unwillingness to compromise. Despite a rose-bud-thorn activity with the professor, the tension persisted. During this period, the team member repeatedly was absent from group meetings and did not contribute to the work, so I took on a larger share of the work to keep the project on track. Along with the other mediator, I led multiple conversations attempting to collaborate effectively and resolve tension. Ultimately, that person left the team, and the remaining three of us continued, with me taking the lead to ensure our work aligned with our overarching goals.
Transitioning from 4 to 3 team members, I led the project to ensure quality and goal alignment.
Transitioning from 4 to 3 team members, I led the project to ensure quality and goal alignment.
Speed Dating with Storyboards
Speed Dating with Storyboards
With a clear understanding of our target audience and their needs, we began brainstorming potential solutions. We were encouraged by the teaching staff to take risks and push the boundaries of social comfort. To spark creativity, we applied the Crazy 8s method, after which each team member created three storyboards addressing a unique need. Additionally, we developed cover pages for each storyboard, outlining the addressed user need and including discussion questions to gain further insights during user testing.
We conducted speed dating sessions with three participants to understand social boundaries in addressing GenAI biases. Key insights included the importance of transparency and accountability for increasing trust in AI systems and a lack of motivation to report biases.
Effective solutions must offer accessible learning of AI biases and establish bias reporting incentives.
Speed Dating with Storyboards
With a clear understanding of our target audience and their needs, we began brainstorming potential solutions. We were encouraged by the teaching staff to take risks and push the boundaries of social comfort. To spark creativity, we applied the Crazy 8s method, after which each team member created three storyboards addressing a unique need. Additionally, we developed cover pages for each storyboard, outlining the addressed user need and including discussion questions to gain further insights during user testing.
We conducted speed dating sessions with three participants to understand social boundaries in addressing GenAI biases. Key insights included the importance of transparency and accountability for increasing trust in AI systems and a lack of motivation to report biases.
Effective solutions must offer accessible learning of AI biases and establish bias reporting incentives.
The Solution
The Solution
After analyzing the speed dating results, we determined that a game highlighting the inner workings of AI would be the most effective solution. A discussion with our TA further refined this idea, leading us to focus on the concept of guardrails—content filters designed to mitigate biased results. We created a low-fidelity paper prototype of the game to simulate the mechanics and conducted usability testing with three participants to assess its effectiveness.
Usability testing revealed that while participants enjoyed the gamified format, engagement was significantly lower than anticipated. After discussing the results with our TA, we established next steps to increase the game's effectiveness, including integrating it with existing AI safety initiatives, such as those by Effective Altruism, or partnering with hackathons and red-teaming events sponsored by companies like Amazon and Meta. Additionally, enabling participants to create prompt-based guardrails or curate datasets to develop fairer guardrails could further engage users in understanding and applying the guardrails to mitigate biases.
The Solution
After analyzing the speed dating results, we determined that a game highlighting the inner workings of AI would be the most effective solution. A discussion with our TA further refined this idea, leading us to focus on the concept of guardrails—content filters designed to mitigate biased results. We created a low-fidelity paper prototype of the game to simulate the mechanics and conducted usability testing with three participants to assess its effectiveness.
Usability testing revealed that while participants enjoyed the gamified format, engagement was significantly lower than anticipated. After discussing the results with our TA, we established next steps to increase the game's effectiveness, including integrating it with existing AI safety initiatives, such as those by Effective Altruism, or partnering with hackathons and red-teaming events sponsored by companies like Amazon and Meta. Additionally, enabling participants to create prompt-based guardrails or curate datasets to develop fairer guardrails could further engage users in understanding and applying the guardrails to mitigate biases.
My Learnings
My Learnings
1.
Effectively applying many UX research methods
Effectively applying many UX research methods
Before this project, I had a foundational understanding of UX research through previous coursework and other projects. This project challenged me to explore a variety of methods and critically evaluate how each could contribute to our goals and deepen our insights within the problem space.
2.
2.
Managing conflict
Managing conflict
This project introduced me to the rose-bud-thorn method as an approach to conflict management. I also developed my communication and mediation skills, facilitating open discussions to navigate tensions and maintain focus on our objectives.
Next Steps
Developing IlluminAItion
As one of my team members and I both have significant programming experience, we will collaborate on developing the game using React and HTML/CSS.
2.
Collaborating with organizations
Once the game is developed, we will present it to existing groups such as Effective Altruism to pair with their existing efforts. We will also share the game with our networks, increasing overall awareness around the concept of guardrails and their applications by doing so.
My Learnings
Effectively applying many UX research methods
Before this project, I had a foundational understanding of UX research through previous coursework and other projects. This project challenged me to explore a variety of methods and critically evaluate how each could contribute to our goals and deepen our insights within the problem space.
2.
Managing conflict
This project introduced me to the rose-bud-thorn method as an approach to conflict management. I also developed my communication and mediation skills, facilitating open discussions to navigate tensions and maintain focus on our objectives.
Next Steps
1.
Developing IlluminAItion
As one of my team members and I both have significant programming experience, we will collaborate on developing the game using React and HTML/CSS.
2.
Collaborating with organizations
Once the game is developed, we will present it to existing groups such as Effective Altruism to pair with their existing efforts. We will also share the game with our networks, increasing overall awareness around the concept of guardrails and their applications by doing so.