This research explores factors influencing U.S. college students’ attitudes and behavioral intentions toward generative AI adoption, using ChatGPT as a case study. Despite divergent discussions on its application and impacts, few studies have investigated people’s perceptions of similar technologies. Given that college students are a potential major user group and the future workforce, this research fills the gap by incorporating the Technology Acceptance Model (Davis, 1989) and the Theory of Planned Behavior (Ajzen, 1991) to explore the relationship between the DVs (attitude and intentional behaviors) and IVs (E.g., perceived usefulness, perceived ease of use, subjective norm, perceived intelligence) in ChatGPT adoption. The study utilized a pre-post single-condition experiment and found a significantly positive relationship between college students’ perceived usefulness and behavioral intentions toward ChatGPT. Even though no statistically significant relationships were found among other variables, a trend where a greater extent of agreement with the statements of the IVs corresponded to higher scores for the DVs was observed. The study contributes to the research on the Technology Acceptance Model in the realm of generative AI adoption and serves as a model for future scholars exploring the adoption of other similar AI tools or for a broader population.

With the skyrocketing advancement of artificial intelligence (AI) and natural language processing (NLP), more sophisticated large language models (LLM) and generative AI tools have emerged (Ray, 2023). Generative AI is capable of generating new data by learning patterns from existing data (Ray, 2023). Notably, ChatGPT, an AI language model developed by OpenAI, is the most prevalent and powerful one in the current market. It utilizes vast databases to generate human-like responses and perform high-level cognitive tasks, sparking diverse discussions on generative AI’s application and potential impacts across various fields since its launch (Liu et al., 2023; Shen et al., 2023). Educational institutions also have raised concerns about its possible role in facilitating student cheating, while some perceive its transformative potential in higher education. (Susnjak, 2022). However, people’s perception of similar technologies remains relatively unknown. Given that college students are a potential major user group and the future workforce, their insights into AI technologies are representative. Due to the existing gap in research on this demographic’s perspectives on AI tools, this research aims to investigate U.S. college students’ attitudes and behavioral intentions toward generative AI, using ChatGPT as an example. 

Previous scholars suggest that perceived usefulness and perceived ease of use influence people’s attitudes and behavioral intentions toward new technology in the Technology Acceptance Model (Davis, 1989). Additionally, the Theory of Planned Behavior also explores factors that determine individuals’ behavioral intentions: attitude, subjective norms, and perceived behavioral control (Ajzen, 1991). Adopting the two theoretical frameworks, the researcher will include the independent variables of perceived usefulness, perceived ease of use, and subjective norm, while the dependent variables will be people’s attitudes and behavioral intentions toward ChatGPT.

 This research specifically aims to explore factors influencing U.S. college students’ attitudes and behavioral intentions toward ChatGPT through conducting a pre-post single-condition experiment. The paper will include a literature review, methodology, results, and discussion. The following literature review will discuss the developments of the two theories and how they contribute to this research design, specifically the hypothesis and methodology. Ultimately, the research will not only contribute to testing the two established theories within the context of generative AI and LLM but also potentially yield valuable insights into the future acceptance of ChatGPT and similar AI technologies among the new generation.

Incorporating the Two Theoretical Frameworks

ChatGPT is a relatively new technology, and no existing research studies people’s attitudes and behavioral intentions toward it. The Technology Acceptance Model (TAM) and the Theory of Planned Behavior (TPB) can be helpful frameworks to investigate factors that influence people’s attitudes and behavioral intentions toward using ChatGPT, particularly among college students.

Both TAM and TPB are built on the TRA and the idea that intentional behaviors influence an individual’s behaviors. TAM suggests that people’s acceptance of new technology is primarily determined by two factors: perceived usefulness and perceived ease of use. TPB can complement TAM by considering additional factors influencing behavior intentions, such as attitudes, subjective norms, and perceived behavioral control. In addition, ChatGPT is an AI technology. When studying the adoption of AI technology, a new factor is often examined: the perceived intelligence (PI), which is “the ability to acquire and apply knowledge and skills” in the Oxford Dictionary of English. Its 5-item intellectual evaluation scale was developed by Warner and Sugarman (Warner & Sugarman,1986) in existing literature.

Research Design

The researcher conducted an in-person pre-post single-condition experiment for this empirical research paper. This design is chosen because the study aims to explore the relationship between the independent and dependent variables. By measuring participants before and after the experiment or intervention, the researcher could assess changes in the outcome variables. In addition, conducting the experiment in -person would provide a higher level of control over the study and allow the researcher to standardize the experiment conditions. Overall, the pre-post single condition and in-person experiment would allow the researcher to measure the effectiveness of the intervention in a relatively controlled and standardized environment. The study has obtained approval from the Institutional Review Board. 

Participants / Sample / Subjects

The target population for this study is college students aged 18 to 30 in the United States, including individuals of all races and genders. To ensure convenience, the experiment was conducted with college students at Stony Brook University. The researcher anticipated recruiting 100 participants based on a power analysis using the R package pwr, which showed that a minimum of 100 participants are required to detect a power of 0.80 with a small to medium effect size of d = 0.25 (Cohen, 1990).  

The participants for this study were recruited using a combination of snowball sampling, in-class engagement, and social media posts. Initially, potential participants were asked to fill out a Google Form to determine their eligibility for the study. Only college students would be considered, and those who expressed interest were asked to select a timeslot for the in-person experiment within the same Google Form.  

Procedure(s) / Intervention

The experiment took place at Stony Brook University in lab-setting, where participants used computers to access the pre-post surveys and interact with ChatGPT 3.5. Each participant was seated at least six feet apart to ensure privacy and prevent interference with others’ participation. The location chosen for the experiment was the lower newsroom at the library, which was equipped with multiple computers and relevant survey administration software to facilitate the smooth running of the experiment. 

The procedure for the experiment involved multiple sessions, each with a maximum of 10 participants and lasting between 10 to 15 minutes. At the start of each session, participants initially took a pre-survey on Qualtrics, an online survey platform, to measure their attitudes and behavioral intentions towards ChatGPT. This pre-survey should take around three to five minutes to complete. 

After completing the pre-survey, participants were then instructed to interact with ChatGPT by inputting the prompts adapted from a previous research study (Jungherr, 2023) and observing its performance. There were four sets of prompts, and participants were randomly assigned with one set comprising four questions. Each set adhered to the same question structure but differed in the topic: gun control, nuclear war, digital misinformation, and climate change. 

After participants interacted with ChatGPT and observed its performances, they ended with a post-survey. It measures the DVs (attitude and behavioral intention) and IVs (E.g., perceived usefulness, perceived ease of use, perceived intelligence, subjective norm, demographics, and covariates) for another five minutes. 

The primary intervention in this study was the use of the prompts designed for ChatGPT. These prompts were intended to elicit specific responses from the participants that would enable the researcher to measure the variables.

Instruments / Measures

The variables were measured on the Qualtrics. Each variable had its own questionnaire section.

(Artem Podrez)

Given the post-pandemic circumstances, collecting data within a short timeframe posed challenges. In the end, there were 25 participants with no missing values. 

The results indicated that the mean attitude score before the intervention (M = 5.22, SD = 1.057, α = 0.81) is 0.156 higher than the mean attitude score after the intervention (M = 5.06, SD = 0.948, α = 0.81). It indicates that the change in mean attitude scores before and after the intervention is not statistically significant (t = 1.05, p = 0.303). The effect size is 0.210, suggesting a small effect. 

The mean score for behavioral intention before the intervention (M = 4.98, 

SD = 1.557, α = 0.95) is 0.312 lower than the mean score for behavioral intention after the intervention (M = 5.30, SD = 1.364, α = 0.96); this indicates a statistically significant difference (t = -2.70, p = 0.012). The effect size, represented by Cohen’s d, is -0.540, suggesting a moderate effect.  

In addition, the researcher did not find that participants’ attention paid to ChatGPT’s response, demographic variables, or news consumption on ChatGPT’s had positive effects on college students’ attitude or behavioral intention toward ChatGPT as each relationship is not statistically significant as well.  

This study investigated college students’ attitude and behavioral intention toward the adoption of generative AI, using ChatGPT as an example through a pre-post and in-person experiment. Overall, the results revealed a significantly positive relationship between college students’ perceived usefulness and behavioral intention toward ChatGPT. Even though no statistically significant relationships were found among other variables, the researcher still observed a trend where a greater extent of agreement with the statements of the IVs corresponded to higher scores for the DVs.  

While the findings contribute to our understanding of the Technology Acceptance Model 

by demonstrating the statistically significant relationships between perceived usefulness and behavioral intention toward ChatGPT, they also open avenues for future research to explore additional factors influencing college students’ attitudes and behavioral intentions regarding the adoption of generative AI. Future researchers are encouraged to examine other generative AI tools, such as Bing AI or Google Bard, to broaden our understanding of user perceptions and adoptions in the evolving landscape of AI.  

However, it’s essential to note the limitations of the study due to the small sample size and potential flaws in the experiment’s design. A larger sample is needed to test meaningful reports based on a power analysis using the R package pwr. As at least 100 participants are expected, the data is still being collected. Future research can enhance the study by recruiting larger samples, improving questionnaire depth, and employing more neutral prompts for manipulation. Testing the two theories on general adult or larger public samples is also recommended. 

To sum up, this study explores college students’ attitudes and behavioral intentions toward ChatGPT using a pre-post single-condition experiment grounded in the Technology Acceptance Model and The Theory of Planned Behavior. The results demonstrated a moderate increase in participants’ mean behavioral intention after interacting with ChatGPT. Additionally, while this study has its limitations, the findings reveal a statistically significant relationship between perceived usefulness and behavioral intention toward ChatGPT. 

The findings contribute insights to the Technology Acceptance Model in the realm of generative AI adoption and offer a glimpse into college students’ perceptions of generative AI. These results encourage future researchers to explore additional factors that may influence the public’s perceptions of the potential benefits or limitations of AI language models like ChatGPT. As data collection is ongoing, the research design also serves as a model for future scholars conducting similar investigations in the ever-changing realm of generative AI on a broader scale.