Warm tip: This article is reproduced from stackoverflow.com, please click
c# google-chrome-extension javascript selenium selenium-chromedriver

Can you track mouse position on a webpage in Selenium webbrouser? C#

发布于 2020-04-03 23:47:23

I am trying to keep track of mouse coordinates in the Selenium browser.

Selenium has no mouse tracking functions. It only has MoveByOffset(int x, int y) and MoveToElement(IWebElement) function. I need to set custom positions and move to them using the MoveByOffset(int x, int y) function.

Selenium allows usage of JavaScript script, but it seems not to allow running the script in the background and retrieving values from the said background function:

We could run something like:

document.onmousemove = function(e)
{
    var x = e.pageX;
    var y = e.pageY;
};

and return values from it, but it doesn't seem to be possible (explained above).

We could also create a Chrome extension and put the script in it's background.js, but I couldn't find a way to retrieve a value from the extension and to use it in code. Is this possible?

I also tried to keep track of the mouse movement in the C# code, but it doesn't seem to be reliable:

We usually need to scroll the page. It can be done with: js.ExecuteScript("window.scrollBy(0," + scrollDownAmountInPixels + ")");, where js is the IJavaScriptExecutor. But scrolling also moves the mouse. We could update the mouse-y value to the scrolled value, but if we are at the top of the page and we scroll up OR if we are at the bottom of the page and we scroll down - the mouse will recieve wrong coordinates. We could test for things like page height, window's distance from the top of the page (scroll amount) and for window's distance from the bottom of the page (page height - (scroll amount + window height size)), BUT JavaScript likes to return 0 instead of some of these values at some specific page locations. I tried scrolling by pixel to find a location in which JavaScript will not return 0 in the needed values, but it just becomes too unreliable.

I really hope to find some help here. Thanks in advance!

Questioner
Roman
Viewed
125
Roman 2020-02-04 07:52

So I created this class. It keeps track of mouse coordinates, but sometimes it can mess them up a little (somewhat fixed, it still can).

To use it you have to call the NewPage() whenever you load up a new webpage, BUT only after it is fully loaded in. Then you can use MoveTo(), passing x, y coordinates or IWebElement to move the cursor to it's location. You can use Scroll() if you need to scroll the page by y pixels. And you can click with Click().

Here's the code:

public class MouseMover
        {
            public int PageHeight { get; private set; }
            public int PageScrolled { get; private set; }
            public int WindowHeight { get; private set; }
            public int MouseXCurr { get; private set; } // current mouse x position
            public int MouseYCurr { get; private set; } // current mouse y position

            private int MouseXToMove { get; set; }
            private int MouseYToMove { get; set; }
            private ChromeDriver Chr { get; set; }
            private Actions click;

            public MouseMover(ChromeDriver chr)
            {
                MouseXCurr = 0;
                MouseYCurr = 0;
                MouseXToMove = 0;
                MouseYToMove = 0;
                Chr = chr;
                NewPage();
                click = new Actions(chr);
                click.Click().Build();
            }

            public void NewPage() // call this whenever entering a new page and it is fully loaded in
            {
                IJavaScriptExecutor js = (IJavaScriptExecutor)Chr;

                Int64.TryParse(js.ExecuteScript("" +
                "var body = document.body," +
                "    html = document.documentElement;" +
                "" +
                "var height = Math.max(body.scrollHeight, body.offsetHeight," +
                "                         html.clientHeight, html.scrollHeight, html.offsetHeight);" +
                "return height;").ToString(), out long ph);
                PageHeight = (int)ph;

                if (PageScrolled != 0)
                {
                    MouseYCurr -= PageScrolled;
                }
                PageScrolled = 0;

                Int64.TryParse(js.ExecuteScript("return window.innerHeight;").ToString(), out long wh);
                WindowHeight = (int)wh;
            }

            public void Click() // click at the current mouse position
            {
                click.Perform();
            }

            public void Scroll(int y) // scroll the page by y pixels down (negative to scroll up)
            {
                IJavaScriptExecutor js = (IJavaScriptExecutor)Chr;
                int oldScroll = PageScrolled;
                if (y > 0)
                {
                    if (PageScrolled + WindowHeight + y <= PageHeight)
                    {
                        js.ExecuteScript("window.scrollBy(0," + y + ")");
                        PageScrolled += y;

                        // sometimes the ScrollHeight gets messed up. This helps to fix it, but it doesn't always fix it
                        Int64.TryParse(js.ExecuteScript("return (window.pageYOffset || document.documentElement.scrollTop)  - (document.documentElement.clientTop || 0);").ToString(), out long s);
                        if (s != 0 && PageScrolled != (int)s)
                        {
                            PageScrolled = (int)s;
                        }

                        MouseYCurr += PageScrolled - oldScroll;
                    }
                    else
                    {
                        if (PageHeight != PageScrolled + WindowHeight)
                        {
                            js.ExecuteScript("window.scrollBy(0," + (PageHeight - (PageScrolled + WindowHeight)) + ")");
                            PageScrolled += (PageHeight - (PageScrolled + WindowHeight));

                        // sometimes the ScrollHeight gets messed up. This helps to fix it, but it doesn't always fix it
                            Int64.TryParse(js.ExecuteScript("return (window.pageYOffset || document.documentElement.scrollTop)  - (document.documentElement.clientTop || 0);").ToString(), out long s);
                            if (s != 0 && PageScrolled != (int)s)
                            {
                                PageScrolled = (int)s;
                            }

                            MouseYCurr += PageScrolled - oldScroll;
                        }
                    }
                }
                else
                {
                    if (PageScrolled >= -y)
                    {
                        js.ExecuteScript("window.scrollBy(0," + y + ")");
                        PageScrolled += y;

                        // sometimes the ScrollHeight gets messed up. This helps to fix it, but it doesn't always fix it
                        Int64.TryParse(js.ExecuteScript("return (window.pageYOffset || document.documentElement.scrollTop)  - (document.documentElement.clientTop || 0);").ToString(), out long s);
                        if (s != 0 && PageScrolled != (int)s)
                        {
                            PageScrolled = (int)s;
                        }

                        MouseYCurr += PageScrolled - oldScroll;
                    }
                    else
                    {
                        js.ExecuteScript("window.scrollBy(0," + -PageScrolled + ")");
                        PageScrolled -= PageScrolled;

                        // sometimes the ScrollHeight gets messed up. This helps to fix it, but it doesn't always fix it
                        Int64.TryParse(js.ExecuteScript("return (window.pageYOffset || document.documentElement.scrollTop)  - (document.documentElement.clientTop || 0);").ToString(), out long s);
                        if (s != 0 && PageScrolled != (int)s)
                        {
                            PageScrolled = (int)s;
                        }

                        MouseYCurr += PageScrolled - oldScroll;
                    }
                }
            }

            public void MoveTo(IWebElement el) // move to the middle of the given web element
            {
                MoveTo(el.Location.X + el.Size.Width / 2, el.Location.Y + el.Size.Height / 2);
            }

            public void MoveTo(int x, int y) // move to the given page coordinates
            {
                MouseXToMove = x - MouseXCurr;
                MouseYToMove = y - MouseYCurr;
                Move();
            }

            void Move()
            {
                bool retry;
                do
                {
                    try
                    {
                        retry = false;
                        Actions actions = new Actions(Chr);
                        actions.MoveByOffset(MouseXToMove, MouseYToMove).Build().Perform();

                        // this will only be executed when the target coordinates enter the screen
                        MouseXCurr += MouseXToMove;
                        MouseYCurr += MouseYToMove;
                        MouseXToMove = 0;
                        MouseYToMove = 0;
                    }
                    catch
                    {
                        retry = true;
                        if (MouseYToMove > 0)
                        {
                            int oldScroll = PageScrolled;
                            Scroll(50);
                            MouseYToMove -= PageScrolled - oldScroll;
                        }
                        else
                        {
                            int oldScroll = PageScrolled;
                            Scroll(-50);
                            MouseYToMove -= PageScrolled - oldScroll;
                        }
                    }
                }
                while (retry == true);
            }
        }