Case Study: Google Gemini Pilot

Embracing GenAI: Harnessing Gemini for Greater Productivity

90-Day Google Gemini Advanced Pilot Demonstrates Value of GenAI in the Workplace

What You’ll Learn

• How a data-driven pilot can demonstrate the ROI of GenAI in workplace productivity.
• The framework for effectively engaging customers and garnering support.
• Staffing considerations and responsibilities to ensure pilot success.

Our Hypothesis

The Governor’s Office of Information Technology (OIT) recognized GenAI’s potential to enhance efficiency and foster innovation in state government, and that Gemini Advanced (Gemini) was a natural tool to implement since the state runs on Google Workspace. But OIT needed a mechanism to effectively evaluate its use and support the tool’s implementation. 

Our Solution

OIT initiated a 90-day pilot in summer 2024 to determine if offering the Gemini GenAI tool would improve workplace productivity and, therefore, yield a positive return on investment (ROI).

The pilot consisted of 150 participants across 18 state agencies who evaluated the ease of use, accuracy, productivity improvement, security implications and innovation potential associated with using a GenAI tool. Leveraging a thriving AI Community of Practice, a dedicated group of testers was created to explore Gemini’s capabilities. These pilot participants not only learned the practical applications of GenAI while developing needed protocols for ethical and responsible use in a safe, facilitated environment, but also fostered an unexpected outcome: a thriving AI community comprised of state employees who still meet to this day as GenAI trailblazers across agencies.

Our Results

The Gemini pilot, designed to explore Gemini’s capabilities and establish ethical protocols, yielded remarkable results captured through a standing survey that participants committed to completing at least three times a week. Over 2,000 surveys were submitted, showcasing significant improvements in productivity, creativity, literacy and human impact. Pilot participants reported the following:

• 74% experienced increased productivity.
• 69% felt less stress due to task and communication support. 
• 73% were able to focus on higher priority work.
• 75% realized enhanced creativity.
• 25% felt greater inclusion and less isolation.
• 83% noticed improved work quality.
• 49% experienced reduced work stress overall.
• 70% had more confidence in their abilities.
• 31% freed up time to upskill their capabilities.

These metrics prove the pilot’s success in empowering state employees and serve as a testament to the potential of responsible GenAI in driving positive change. Colorado is now equipped to build upon these learnings and responsibly implement the technology.

Our Approach

The success of our Gemini Advanced pilot was built on a structured and comprehensive approach. Below is an outline of our step-by-step approach to developing and evaluating a GenAI pilot program before statewide rollout.

Step-By-Step

Step 1 | Choose a Tool

OIT selected Gemini Advanced due to the following factors:

• Seamless integration with existing Google Workspace applications.
• Familiarity and ease of adoption for state employees.
• Security aligns with the existing state IT Acceptable Use Policy.
• Procurement terms were approved, and vendor support was included in the contract.

Step 2 | Develop Communication Plan

A strong communication plan ensured all stakeholders were informed, aligned and engaged throughout the GenAI pilot, preventing confusion and fostering successful implementation:

• Weekly emails set expectations for topics covered during the CoP and trainings, including where to find information and Q&A.
• A central online hub included a training calendar, links to the attestation and external training, and past recordings. This website allowed participants to have a one-stop shop for finding information about the pilot.
• A chat room allowed participants to continue discussing findings and asking questions between CoP meetings and trainings. This chat space is still active and serves as a buzzing water cooler for Gemini users in the state.

Step 3 | Recruit

Tell people you’re running a Gemini pilot, and ask them to sign up! 

• SAMPLE: Gemini Pilot Sign-up

Step 4 | Gather Attestations

Once your pilot list is finalized, have selected participants fill out an attestation that meets your security standards. An attestation ensures participants understand and agree to legal, ethical and security guidelines, promoting responsible use and protecting the organization from potential risks. Essentially, it's a formal acknowledgment of understanding and commitment to safe and appropriate GenAI interaction.

• SAMPLE: Pilot Participation Agreement

Step 5 | Require Training

Have participants complete a GenAI Literacy training that helps them understand how to use the tool correctly, safely and ethically, maximizing the pilot's effectiveness and minimizing potential risks. 

• SAMPLE: We went with InnovateUS Responsible AI for Public Professionals: Using GenAI at Work due to InnovateUS’ extensive experience with state and local governments, accessibility and strong expertise in AI.

Step 6 | Grant Access

When participants can show a completion certificate for training, give them access to Gemini. 
Participants send a screenshot or a PDF version of the completion certificate to the Pilot team.

Step 7 | Collect & Track Data

With attestations and training now complete, participants have access to the tool. You can now start collecting and monitoring their engagement. 

• Surveys
  • Participant surveys containing the same data fields were collected as a feedback mechanism. Using the same survey throughout the pilot allowed the team to track progress, identify potential issues, and understand Gemini’s impact. Participants were asked to complete the survey at least three times a week as they tested the GenAI tool.
  • SAMPLE: Gemini Weekly Survey
• Meeting attendance
  • Weekly pilot Community of Practice (CoPs) meetings and trainings were hosted virtually. The roster was saved so that organizers could note attendance and get a headcount of active participants.
• Email open rate
  • Communication was sent via an email management platform to track open and click-through rates.
• Website visits 
  • Analytics were monitored to track website activity and engagement on the central online hub.

Step 8 | Host Learning Cohort

Lead a regularly standing learning cohort to reinforce the responsible use of GenAI and create a collaborative community 

• Examples:
  • Hold weekly community of practice sessions with participants
  • Host trainings that cover how to work with Gemini, prompt ideas and Q&A

Step 9 | Analyze Data

At the close of the pilot, analyze the data to determine if Gemini is a good fit for your agency.

• Data can come from survey responses, email/webpage engagement, etc.

Step 10 | Create a Rollout Plan

If you decide Gemini is a fit, it’s time to create a rollout plan for your agency. 

• Example: Communication toolkit

Evaluation Method

Pilot participants were encouraged to complete a standing survey at least thrice weekly while using the GenAI tool  to evaluate the following areas:

• Ease of Use
• Productivity
• Innovation & Creativity
• Fairness & Bias
• Explainability & Interpretability
• Validity & Reliability
• Accountability & Responsibility
• Privacy & Data Protection

Resources

• Use Cases
• Presentation
• Sample Survey Questions