desktop-control

Use the DesktopController class wrapping PyAutoGUI: call click(x, y) for clicks, type_text("hello", wpm=60) for human-like typing, and hotkey('ctrl', 'c') for shortcuts. Initialize with DesktopController(failsafe=True) so moving the mouse to a screen corner aborts execution.

Use find_on_screen(image_path, confidence=0.8), which relies on OpenCV template matching via PyAutoGUI's locateOnScreen. It returns the match's x, y, width, and height, so you can click the center coordinates of the found element.

Yes, but secondary displays may use negative coordinates since screen coordinates start at (0, 0) on the primary monitor's top-left corner. Windows DPI scaling can also shift coordinate accuracy, so verify positions with get_mouse_position() and get_screen_size().

Some applications, especially games and secure software, block synthetic input events. Ensure the target window has focus using activate_window(), try increasing the typing interval, and run Python with administrator privileges where required.

The AIDesktopAgent currently uses rule-based pattern matching rather than a real LLM, so it only handles recognized patterns like drawing in Paint, typing in Notepad, and launching apps. Screen analysis lacks OCR and object detection, and game playing is experimental.