TLDR: Part 2 – The bottom line is I built a custom solution using a low-cost gaming Tobii hardware eye tracker, Microsoft Surface tablet and using software built as a research project from a university in Germany (Gaze the web browser) along with WordPress… is it perfect? no, does it work? Yes, and pretty well – check the video.[ADD LINK]
Reality check time… ICUs and patients in these situations are not the best places to test things… whilst this ‘could’ work on an ongoing basis, the reality is the environment, staff changes, and position of the patient don’t lend well to eye-tracking of this type, see more in the post.
If you haven’t read part one then you should, it will give some context to what this is about.
I don’t really expect anyone to read these long posts about how I research and formulate opinions on things, but If you do then thanks 🙂 you have a friend over here.
I’ve researched the software and hardware involved in eye-tracking and its ability to enable people with motor disabilities a way to communicate.
What I will need for this to work:
- Tablet Device (laptops have a keyboard that gets in the way) – capable of running the software
- Eye-tracker – something that is not going to cost the earth to test and is compatible with the software
- Stand, holder – this needs to be flexible in terms of adjust-ability in height and angle and manageable in an ICU setting
- Operating system – compatible with the software chosen
- Open source or paid s/w options, depending on compatibility with hardware and OS
- Emulated pointers or purpose-built applications, web-based apps?
After a lot of digging around with the various hardware software compatibility lists, I focused on the following hardware/software options.
Hardware – Purchased
Microsoft Surface Pro Tablet – this is slim, powerful and has the most compatibility with both hardware and software I found.
- Microsoft Surface Pro 4 256GB SSD, 8GB RAM, i5-6300U – Second Hand (£165)
This came and was delivered quickly, it was supposed to have some faulty touch component with the screen but I’ve found no fault with it and I actually really like it as a device.
Tobii Eye Trackers – these again appear to have the best reputation and are sold as individual units.
- Tobii 5 Gaming Eye Tracker (£160) – Second hand, this looked to me like a good starting point and cheap.
I purchased this second hand from someone who used it in a flight simulator setup, it arrived quickly and was in perfect condition.
There were some issues after I installed the software due to calibration of the device, it just wasn’t picking up the gaze points correctly and I thought the unit may have been faulty. The calibration software that comes with the ‘gaming’ version of this device is pretty crap, certainly not like their AAC device calibration software from what I found.
Searching for some advice on the Tobii website about calibration of the gaming device didn’t yield much, however, you can install calibration software for one of their other devices which is WAY better and I use this all the time now to get the calibration perfect AND it is perfect.
The Tobii 5 eye-tracker is responsive, accurate and gets to specific points on the screen very well. It is also small and looks good with a solid feel. – great bit of kit.
One thing to note with these devices and this goes for people using it for gaming – HEAD POSITION – this matters, always re-calibrate before you operate as it takes 30 seconds and makes it so much more accurate. I don’t think the head position is mentioned enough when you are new to eye-tracking tech when it comes to the accuracy of the gaze points, especially when you are trying to click on relatively small things.
Stand – must be suitable for ICU, easy to use and have height and angle adjust-ability
- Option one – Tripod mount for Surface/Tablet (£10)
This is OK but feels a little unstable if placed on a table and wouldn’t want to use it in an ICU hospital department that way. Great if you are sitting on the lounge though.
- Option two – ZenCT Tablet Stand Clamp Mount Holder (£51)
This is a great bit of kit and has a free lifetime warranty on all of the components, is very robust and holds the tablet well. However, does drop a little bit over time but I just may need to tighten the ball grip harder with my tiny hands and the Surface is quite a heavy device with an eye tracker mounted to the front.
One thing that is a little tricky with this setup and is worth mentioning in case anyone else tries this at home, fixing it all together…
The Tobii eye tracker comes with a bunch of little bits to attach to monitors and other surfaces, however, in my case this didn’t work with the Surface and the ZenCT stand.
I initially used blue-tack to attach it as a temporary measure but after a while, with the heat, it fell off.
Here is what I made, it is not pretty but does the job for now [insert picture.]
The Tobii eye-tracker has a little grove in the back with a magnet, and you get a magnet that fits in this groove in the box that has some double-sided tape. Using this and a small piece of cut plastic mounted under the tablet seems to work, it’s not pretty but functional.
Total hardware cost: £368
Software – Evaluating
Microsoft Windows for the OS – this has the largest number of hardware and software compatibility (comes with MS Surface Tablet).
- Grid 3 by SmartBox (2-month trial)
- Communicator 5 by Tobii Dynavox (1-month trial)
- Talon (this didn’t work with my hardware)
I did a bit of research to see if there had been any specific web application (run in a browser) that worked with eye-trackers through some kind of browser plugin or downloadable element but I couldn’t find anything specific.
I understand why most AAC is designed to run locally as it’s often the only method of communication for the user and the internet can be dodgy in certain circumstances but this is changing and will soon be as necessary and regular as power.
What I was hoping for was some library that worked with page markup, e.g I marked up a div with say “data-gaze-click” and css “gaze-hover-pulse” for example. This would then work with the eye trackers to create a gaze pulse effect on the element and would invoke a click option after a pre-configured gaze duration alongside the underlying divs href link.
With this kind of library and markup, you could in theory start to create interesting web apps that use the eye-tracker tech on a standard web page for anyone with the hardware to use.
I would hope this eventually becomes part of modern web browser accessibility and the use of eye-trackers as inputs with a framework to assist with control.
However, I did find GazeTheWeb which is a university project out of Germany specifically for enabling the ‘whole’ web for those that are motor impaired using their eyes or voice which I added to the list to have a look at in more detail.
This was installed without fault and I configured the Tobii eye-tracker pretty easily and a 60 day trial of the software by default.
The UX and UI of this software are actually really nice, from browsing around the web this is distinctly lacking in this category of software.
This product is actually a mini-suite of applications, each of them with a user type in mind or a function that is needed – what’s in the box?
- Environmental control – turn on your TV, lights, fan, answer the intercom etc if you can have it connected up.
- Computer control – mouse and keyboard emulation.
- Multiple grid-based communication templates for multiple types of users and education levels, such as symbols, words, phrases with images.
- Speech synthesis
- Apps – such as Spotify, Facebook, Whatsapp etc
- Mobile control, text and phone calls, email
- Grid editing and command action buttons
- Eye-tracker calibration in-app
Grid 3 not only looks good but it works really wee too, I have no question why Tobii wanted to keep them.
I have barely used the grid creation and editing solution but it could well be what I need moving forward and as soon as I have had a look at the rest I’ll come back to see what I can build.
I downloaded an installed Communicator 5 by Tobii and it all worked as expected with no obvious issues.
First impressions, it’s teal, I have never been a fan of teal and the overall inial grid that pops up feels dated with tiny fonts on the grid items which dont seem very organised.
Now maybe I was given a good thing too early with Grid 3 but from the same relative price… I like SmartBox. However, it’s all about the functionality and what it can do for my use case.
So what’s in the box?
This is a little different to the rest, it’s not a commercial product and it’s not really a solution in its current form, however, it gave me a few ideas.
see the research paper here
There are pros and cons to all of the software solutions above, but overall non were designed for my use case although some had the potential to be.
I knew mum wouldn’t want anything complicated, it would need to be specific to her actual current needs. For example, she does not need to go on to Facebook, browse the web or ask complex questions, all she really needs to do is communicate her basic situational needs.
Insight into the eye tracker market, some gripes and opinions
I have only been researching this market for a few weeks, however, it appears to me this is the holy grail for marketers and human behaviour research among many other things.
There are two parts to this:
Usage of eye gaze tech as a hands-free input to free up the hands for other things, or to keep them doing other things in the case of driving.
Naturally, eye trackers are useful to people who can use their hands to operate standard computing input methods and have been around longer than voice control from what I understand.
There has been a lot of interest from Apple, Microsoft, Google and Facebook in this technology in recent years almost all purchasing a company that works in this industry. From what I gather this could be an important piece in the desire for all these companies to move properly into augmented reality. If you think about it you don’t carry a keyboard and mouse with you and there will need to be some interaction with your augmented environment.
I believe the able-bodied eyes should be used in consort with other input methods in modern computing, for example when browsing you could use the eyes to scroll the page or turn a page in a book. I don’t feel that you would want to type (and I have tried) with your eye for a long period, although you could.
So, how do I see this working?
Once you are used to it it is actually way faster to navigate than using a mouse, the only issue able-bodied and verbal users are the speed of ‘click’s when you know you want to engage with something. ADD MORE
This is all about saving the effort with mouse position
Dropdown menus are easy to navigate with eyes after a right-click
The biggest issue with external eye trackers at the moment is head position, when this is not the same as the previously calibrated position the eyes don’t track on the screen position correctly. This is why calibration each time is necessary, once they crack this you’ll find them everywhere as an input device – shopping centres, cars etc and with augmented.
Where the eye looks is critical to decision making and marketers are using this info with creating better advertising to sell shit, they love it.. it works.
The eyes are the windows to the soul as well as what you are about to do next!
Google did a research paper back in 2007 about the search engine result pages
I don’t want this to come across as a long diatribe into your business, I respect your tech but something doesn’t pass the pub test.
Started in 2001 a startup of geeks in Sweeden decided to start a company in eye-tracking hardware, they created a few different aspects of the business. The first aspect was the tech for researchers, the second was the one for ACC to assist with disabilities, third is the ‘tech’ part which basically is everything else including gaming. They have acquired a few businesses and have quite the meteoric rise from what I can see, an impressive company with awesome tech.
https://www.youtube.com/watch?v=QUuiON9cfaM <– they really want to get gaming working with eye-trackers as it’s a massive market, I can see why, however, this bring me to something that makes me a little uncomfortable about them.
Now Tobii, if you ever read this (you won’t), I noticed this (Smart Box used to be owned by Tobii)
“From August 2021, Tobii gaming cameras will no longer be compatible with AAC software including their own titles and Grid 3.
This is because Tobii Dynavox want to ensure that their eye tracking solutions used with AAC software can be supported effectively.
This means that any new gaming cameras, such as the recently released Tobii Eye Tracker 5, will not be supported in AAC software – including Grid 3.”
Hmm, considering the gaming variety is 10% of the cost of the AAC version (at the lower end) than this to me is a company who suddenly realised their gaming device is actually suitable for their AAC market division. I suspect if they tried to sell a device for a 1000+( PCEye 5) to the gamers without a real need, then that wouldn’t happen eh… however, those that do need it most and could get it cheaper if it works (and it does, I’ve tested) will pay a fortune more.
I am sure there are a bunch of reasons why the more expensive ones are better etc… but practically, for the price?
I’ve not yet read the financials to see if there is a breakdown of product sales (public company), but I suspect they make a lot from the sales of AAC devices.
Especially as people that need them often get charitable grants to help them pay for them to ease the burden.
I get the need for profits, and I get the need for reliable accessories in the market but I think something here feels off.
Geek Alert – they are actually breaking Google advertising T&Cs by running ads targeting the same keywords with different URLs (business units) and belonging to the same company. This allows them to take up a lot of page one real estate dominating the lead gen and a little anti-competitive in Googles eyes.. watch out guys, you may get banned! (I am sure they have some smart technical loophole that they are registered as separate businesses etc, however, it’s the same company).
http://2018.petmei.org/ ← eye tracker thing.
USB-A to USB-C adapter
Amazon Talking … and Downloading…
GRID 3 getting started