y coordinate issue

[daskabe]

Description

i have total of 3 monitors(including my mac laptop), moved the cursor to one of the externalmonitors with shown configuration.
For some reason the Y coordinate seems to change between clicks.

with shown configuration i was expecting clicks at the same spot

Version

1.7.24

Steps to Reproduce

run application with shown configuration while using multiple screens

Screenshots

Device

macbook pro

Operating System

15.2 (24C101)

Additional Context

No response

[daskabe]

accidentally closed it but this is still issue

[othyn]

Thanks for submitting the issue! I can't guarantee when I can get around to a fix, but I'm open to PR's! This also appears related to #77

[daskabe]

I am working to resolve it but not working so far.

When i put the cursor at (0,0) ; bottom left of my macbook screen; the reported click location in the logs is at (0,1116)..which the click works as expected. But this wont work when you have multiple screens since NSScreen.main!.frame.height is not dynamic to figure out which screens frame height to subtract from. also i dont know enough about ios regarding why we subtract y location from frame height.

problem code here

        let mouseX = self.mouseLocation.x
        let mouseY = NSScreen.main!.frame.height - self.mouseLocation.y. <---

here is more detail i am trying to work through from chatGpt regarding global coords etc.

NSEvent.mouseLocation reports the current global position of the mouse in screen coordinates. The values are relative to the macOS main screen’s coordinate system.

---

Details about NSEvent.mouseLocation

1. Global Screen Coordinates:

   • The position is reported in the coordinate space of the primary screen (NSScreen.main).
   • The origin (0, 0) is at the bottom-left corner of the primary screen.

2. For Multiple Screens:

   • The coordinates extend into negative or positive space depending on the arrangement of secondary monitors.
   • For example:
     • If you have a monitor to the left of the main screen, the X-coordinates can be negative.
     • If you have a monitor above the main screen, the Y-coordinates can be positive.

3. Y-Axis Inversion:

   • In NSEvent.mouseLocation, the Y-axis increases upward, which is different from the coordinate system used for NSScreen or NSView where the Y-axis increases downward in a flipped coordinate system.

---

Example: Multi-Screen Setup

Screen Arrangement:

• Main Screen (1920x1080): Origin at (0, 0).
• Secondary Screen (to the left, 1280x720): Origin at (-1280, 0).
• Secondary Screen (above, 1920x1080): Origin at (0, 1080).

NSEvent.mouseLocation Example Values:

• Mouse on Main Screen (center):

  NSEvent.mouseLocation: (960, 540)
  

• Mouse on Left Screen:

  NSEvent.mouseLocation: (-640, 360)
  

• Mouse on Top Screen:

  NSEvent.mouseLocation: (960, 1620)
  

---

Converting to Screen-Relative Coordinates

To convert NSEvent.mouseLocation into coordinates relative to a specific screen:

1. Identify the screen containing the mouse location:

   let mouseLocation = NSEvent.mouseLocation
   if let screen = NSScreen.screens.first(where: { $0.frame.contains(mouseLocation) }) {
       print("Mouse is on screen: \(screen)")
   }

2. Adjust the coordinates relative to the screen’s frame:

   let screenFrame = screen.frame
   let screenRelativePoint = CGPoint(
       x: mouseLocation.x - screenFrame.origin.x,
       y: mouseLocation.y - screenFrame.origin.y
   )
   print("Screen-relative point: \(screenRelativePoint)")

---

Practical Uses

1. Detecting Mouse Movement:

   • Track the mouse position globally and determine which screen it’s on.

2. Creating Tools for Multi-Screen Environments:

   • Adjust mouse actions (like clicks or drags) based on the screen the mouse is currently on.

3. Screen Captures:

   • Convert the global position to a specific screen for capturing a portion of the screen.

---

[othyn]

Super! That's quite handy debugging. So looks like the API it's using to get the coord of the mouse only applies to the main screen, which makes sense as to why its now causing issues on alternate displays.

Again, can't guarantee when I can get around to looking into it, but that's really handy for pointing me in the right direction. Thanks!

[canesaccount]

I am working to resolve it but not working so far.

When i put the cursor at (0,0) ; bottom left of my macbook screen; the reported click location in the logs is at (0,1116)..which the click works as expected. But this wont work when you have multiple screens since NSScreen.main!.frame.height is not dynamic to figure out which screens frame height to subtract from. also i dont know enough about ios regarding why we subtract y location from frame height.

problem code here

        let mouseX = self.mouseLocation.x
        let mouseY = NSScreen.main!.frame.height - self.mouseLocation.y. <---

here is more detail i am trying to work through from chatGpt regarding global coords etc.

NSEvent.mouseLocation reports the current global position of the mouse in screen coordinates. The values are relative to the macOS main screen’s coordinate system.

Details about NSEvent.mouseLocation

1. **Global Screen Coordinates**:
   
   * The position is reported in the coordinate space of the primary screen (`NSScreen.main`).
   * The origin `(0, 0)` is at the **bottom-left** corner of the primary screen.

2. **For Multiple Screens**:
   
   * The coordinates extend into negative or positive space depending on the arrangement of secondary monitors.
   * For example:
     
     * If you have a monitor to the **left** of the main screen, the X-coordinates can be negative.
     * If you have a monitor **above** the main screen, the Y-coordinates can be positive.

3. **Y-Axis Inversion**:
   
   * In `NSEvent.mouseLocation`, the **Y-axis increases upward**, which is different from the coordinate system used for `NSScreen` or `NSView` where the **Y-axis increases downward** in a flipped coordinate system.

Example: Multi-Screen Setup

Screen Arrangement:

* **Main Screen (1920x1080)**: Origin at `(0, 0)`.

* **Secondary Screen (to the left, 1280x720)**: Origin at `(-1280, 0)`.

* **Secondary Screen (above, 1920x1080)**: Origin at `(0, 1080)`.

NSEvent.mouseLocation Example Values:

* Mouse on **Main Screen (center)**:
  ```
  NSEvent.mouseLocation: (960, 540)
  ```

* Mouse on **Left Screen**:
  ```
  NSEvent.mouseLocation: (-640, 360)
  ```

* Mouse on **Top Screen**:
  ```
  NSEvent.mouseLocation: (960, 1620)
  ```

Converting to Screen-Relative Coordinates

To convert NSEvent.mouseLocation into coordinates relative to a specific screen:

1. Identify the screen containing the mouse location:
   let mouseLocation = NSEvent.mouseLocation
   if let screen = NSScreen.screens.first(where: { $0.frame.contains(mouseLocation) }) {
       print("Mouse is on screen: \(screen)")
   }

2. Adjust the coordinates relative to the screen’s frame:
   let screenFrame = screen.frame
   let screenRelativePoint = CGPoint(
       x: mouseLocation.x - screenFrame.origin.x,
       y: mouseLocation.y - screenFrame.origin.y
   )
   print("Screen-relative point: \(screenRelativePoint)")

Practical Uses

1. **Detecting Mouse Movement**:
   
   * Track the mouse position globally and determine which screen it’s on.

2. **Creating Tools for Multi-Screen Environments**:
   
   * Adjust mouse actions (like clicks or drags) based on the screen the mouse is currently on.

3. **Screen Captures**:
   
   * Convert the global position to a specific screen for capturing a portion of the screen.

Try #86 should be fixed, though i was only able to test this with two monitors.