Understanding ACCELQ OCR-Based Automation

ACCELQ provides Optical Character Recognition (OCR) capabilities to enable automation for scenarios where traditional locator-based methods fail. This feature allows text-based interaction with any kind of desktop application, including standalone applications, across both Windows and Mac platforms.

By leveraging OCR, ACCELQ can detect and interact with text elements on a screen, making it possible to automate user actions like clicking buttons, entering text in fields, and retrieving visible content, regardless of the application type.

Real-World Applicability

The OCR-based automation capabilities in ACCELQ can be beneficial in various scenarios:

• Automating interactions with applications that do not expose UI elements through traditional locators (e.g., legacy applications, remote desktops, virtualized environments).
• Handling text-based user interfaces where element locators are unstable or unreliable.
• Enabling automation in cross-platform applications where UI structures vary between different operating systems.

Important: Use OCR Capability Sparingly

While OCR-based automation can be a powerful tool, it should only be used as a last resort when no other approach is feasible. Over-reliance on OCR can lead to inaccuracies and increased maintenance efforts due to variability in text detection.

Supported OCR Commands

ACCELQ provides several OCR-driven commands to facilitate automation in such environments. These commands allow users to locate, click, and enter text based on OCR-recognized text elements.

1. Click on OCR-Located Text

This command clicks on a specified text that has been identified using OCR.

• Use Case: Clicking on buttons or labels that cannot be identified through traditional locators.
• Key Parameters:
  • Label text: The text to be clicked.
  • Matching criteria: Defines how the text should be matched (e.g., exact match, contains, ignore case, regex match).
  • Click Position: Allows specifying where on the detected text the click should occur (e.g., center, top-left with offset, bottom-right with offset).
  • Search Area: Defines whether the text should be located anywhere on the screen or near a specific reference text.

2. Double Click on OCR-Located Text

This command performs a double-click action on text located using OCR.

• Use Case: Double-clicking icons, opening files, or expanding options in software interfaces.
• Key Parameters: Similar to the click command, allowing precise targeting and search refinement.

3. Right Click on OCR-Located Text

This command performs a right-click action on the identified text.

• Use Case: Opening context menus or performing right-click actions where traditional UI elements are not accessible.
• Key Parameters: Similar to the click command, including positioning, text matching, and search area.

4. Get All OCR-Located Text

This command retrieves all readable text from the current screen using OCR and provides it in JSON format, including text positions and sizes.

• Use Case: Debugging and identifying how OCR interprets screen content, assisting in creating more precise automation scripts.
• Output: A JSON structure containing detected text and its respective position details.

5. Enter Text in Input Field Based on OCR-Located Label

This command enters specified text into an input field associated with a nearby OCR-identified label.

• Use Case: Automating form filling where input fields lack proper locators but have visible labels.
• Key Parameters:
  • Text to Enter: The text that needs to be inputted.
  • Label Text: The label of the input field, which helps locate the field.
  • Position: Specifies where the click should happen relative to the label before entering text.
  • Search Area: Determines whether to search for the input field anywhere on the screen or in proximity to a reference text.

• Accuracy Variability: OCR-based text detection is not always 100% accurate, especially for low-resolution screens or complex backgrounds.
• Pre-validation: It is recommended to use the "Get All OCR-Located Text" command to verify how OCR interprets the on-screen text before defining automation steps.
• Performance Considerations: OCR scanning may take slightly longer than locator-based identification, so it should be used only when necessary.
• Desktop Automation Only: Ensure OCR-related commands are strictly used for Desktop Automation and not for web automation.
• Active Window Requirement: The target window must be on top and activated to allow accurate text recognition.
• Single Test Execution: Execute only one test case at a time (No Parallel Execution) to maintain proper window focus and avoid conflicts in screen text identification.
• Unlocked Machine: The agent machine should not be locked during execution or in the middle of a test run to ensure smooth OCR-based text detection.