Accessibility Solution Evaluation

About Accessibility Solution Evaluations

For the state of Colorado’s executive branch agencies (except CDE and CDHE), the recent legislation has driven OIT’s Solution Engineering team to design and implement a new service called Solution Evaluations. The Solution Evaluations are designed to provide risk assessments on inbound ICT purchase requests from all agencies. The evaluations will include high-level risk assessments on software, hardware, and solicitations. With each evaluation request, accessibility, security, and contracting will be reviewed and scored based on models developed by the OIT security, accessibility and procurement teams.

If your agency is not an executive branch agency (except CDE and CDHE) then you should perform your own risk assessment to determine the risk in moving forward with a specific vendor. Follow the steps in the Standard Operating Guide to solicit information using the vendor checklist questions, analyze the information provided including what is given in the accessibility compliance report for that product, and score the answers using the High Level Risk Assessment template (Google Sheets).

Reviewing the Accessibility Compliance Report

A Voluntary Product Accessibility Template, or VPAT, is a template designed to help vendors and product developers report the level of conformance for an Information and Communications Technology (ITC) digital product such as a website, application, software, or interface. 

A completed VPAT is a product’s Accessibility Conformance Report (ACR) and should be completed whenever a product has been created or updated/upgraded, and subsequently each year. Vendors should have conformance reports for every digital product they offer. However, if they don’t, this is not an immediate showstopper.

When vetting products, including product upgrades, review all of the success criteria, their conformance levels, and explanations/remarks for each level you need to be compliant with. 

As you do the review:

• Tally the number of partial and non-compliant level success criteria in the report.
• Read the remarks and explanations; especially for partial and non-compliant criteria.
  • Think through the remarks to discern if someone who is blind or low-vision (including color blindness), deaf or hard of hearing, cannot use a keyboard and/or mouse, or has a cognitive disability would be limited in using the product without that functionality being compliant. 
• Check for completion, accuracy and detail in the remarks. 
  • You should have contact information, the name and version of the product and what version and level of the WCAG the report was developed against.
  • You should not see any criteria with a conformance level that is blank. If the criteria is not applicable to the product the report should indicate this.

As you are reviewing the compliance report you should also consider these things:

• Is all of the information in the cover page complete. 
• Ask the vendor who on their team completed the report.
• Ask if you can speak with the development team to gain clarification on any elements in the report. Note that some vendors may not allow this but it’s always good to ask.

Warning Signs

There are a few things that may help you determine whether you will have a good partnership and will be able to work with them to continually address accessibility issues. Note that these warning signs are not necessarily indicators that the product accessibility risk is high but do require more thought, research and discussions with the vendor to determine whether there is warranted concern and/or the risk is too high.

• Vendor asks “What is a VPAT?”
• No VPAT available when accessibility is applicable
• No VPAT; only global, nonspecific accessibility statements
• ACR outdated or using old VPAT 1.0 template)
• Single VPAT representing mixed set of product types / vendor offerings (ex: multiple pass represented)
• Much of vendor ICT accessibility documentation is inaccurate
• Much of the compliance report is not filled in
• Blank fields in title block: missing product name / version, blank or irrelevant response to “Eval Method Used”, date, contact info, etc.
• Large amounts of partially and/or non-compliant success criteria in the compliance report
• Success criteria for critical functionality such as screen reader accessible navigation, accessible forms or other basic functionality required to use the product is not accessible.
• Vendor refuses to agree or sign any contract that includes accessibility requirements or responsibility.
• Vendor is unwilling to provide accessibility documentation such as a compliance report, or they make it extremely difficult to obtain this information.

This is not an exhaustive list of warning signs but gives you an idea of where critical gaps may appear as you are performing an assessment and therefore require more information or discussion on whether it is a high risk to move forward.

What Do the Scores Mean?

Solution evaluations don't necessarily produce an absolute yes or no answer for a product (or development service). It is up to your organization to decide how to proceed when an assessment score is calculated. In most cases, OIT puts the final decision into the hands of the agency to decide whether to move forward. However, it is highly recommended that your organization be thoughtful and critical in your decision, especially when a high risk is identified.

When you do receive a high (or even medium) risk rating, consider these things before moving forward with a decision:

• Are there other products out there that can meet the same need (do the research).
• If this is a sole solution, meaning nothing else is out there, decide whether the solution is critical to your business and/or customers
• If you do need this solution, talk about what kind of accommodations plan you can put in place to meet the needs of people who cannot equitably access the product
• Think about how long you may need the product and make a plan to revisit periodically to determine if accessibility can be improved or if there is a more accessible option to replace the product

Ultimately, the decision is up to your organization and using the risk assessment tools can help you make the most thoughtful and well researched decision you can.