Part 2 TLDR – Microsoft Surface, Tobii gaming hardware eye-tracker and Grid 3 from Smartbox was the winner.
I also had a bit of fun building my web app and using the Gazethweb browser to interact with what I came up with, but ultimately not designed perfectly for the situation or anywhere near as powerful.
ICU environments and patients in them are not the best places to test things as every single day is a challenge.
Intro
If you haven’t read part one then you should, it will give some context to what this is about.
As it turns out this is a very long post and I don’t expect anyone to read it unless they are in a similar position, but if you do, then thank you.
I’ve investigated the software and hardware involved with eye-tracking and its ability to enable people with motor disabilities a way to communicate. I have then built a solution out of my research and applied it to my mums’ situation, where I could.
What I will need for this to work:
Hardware
- Tablet Device (laptops have a keyboard that gets in the way) – capable of running the software.
- Eye-tracker – something that is not going to cost the earth to test and is compatible with the software.
- Stand, holder – this needs to be flexible in terms of adjust-ability in height and angle and manageable in an ICU setting.
Software
- Operating system – compatible with the software chosen
- Open source or paid s/w options, depending on compatibility with hardware and OS
- Emulated pointers or purpose-built applications, web-based apps?
The Options
After a lot of digging around with the various hardware software compatibility lists, I focused on the following hardware/software options.
Hardware – Purchased
Microsoft Surface Pro Tablet – this is slim, powerful and has the most compatibility with both hardware and software I found.
- Microsoft Surface Pro 4 256GB SSD, 8GB RAM, i5-6300U – Second Hand
This was delivered quickly and it was supposed to have a faulty touch screen, but I’ve found no fault with it so far.
Tobii PCEye
Tobii Eye Trackers – these again appear to have the best reputation and are sold as individual units.
- Tobii 5 Gaming Eye Tracker – Second hand, this looked to me like a good starting point and cheap.
I purchased this second hand from someone who used it in a flight simulator setup, it arrived quickly and was in perfect condition.
There were some issues after I installed the software due to calibration of the device, it just wasn’t picking up the gaze points correctly and I thought the unit may have been faulty. The calibration software that comes with the ‘gaming’ version of this device is pretty crap, certainly not like their AAC device calibration software from what I found.
Searching for some advice on the Tobii website about calibration of the gaming device didn’t yield much, however, you can install calibration software for one of their other devices which is WAY better and I use this all the time now to get the calibration perfect AND it is perfect.
The Tobii 5 eye-tracker is responsive, accurate and gets to specific points on the screen very well. It is also small and looks good with a solid feel. – great bit of kit.
Stand – must be suitable for ICU, easy to use and have height and angle adjust-ability
- Option one – Tripod mount for Surface/Tablet (£10)]
This is OK but feels a little unstable if placed on a table and wouldn’t want to use it in an ICU hospital department that way. Great if you are sitting in the lounge though.
- Option two – ZenCT Tablet Stand Clamp Mount Holder (£51)
This is a great bit of kit and has a free lifetime warranty on all of the components, is very robust and holds the tablet well. However, it does drop a little bit over time but I just may need to tighten the ball grip harder with my tiny hands and the Surface is quite a heavy device with an eye tracker mounted to the front.
Assembly
One thing that is a little tricky with this setup and is worth mentioning in case anyone else tries this at home, fixing it all together…
The Tobii eye tracker comes with a bunch of little bits to attach to monitors and other surfaces, however, in my case this didn’t work with the Surface and the ZenCT stand.
I initially used blue-tack to attach it as a temporary measure but after a while, with the heat, it fell off.
Here is what I made, it is not pretty but does the job for now
The Tobii eye-tracker has a little grove in the back with a magnet, and you get a magnet that fits in this groove in the box that has some double-sided tape. Using this and a small piece of cut plastic mounted under the tablet seems to work, it’s not pretty but functional.
Total hardware cost: £368
Software – Evaluating
Microsoft Windows for the OS – this has the largest number of hardware and software compatibility (comes with MS Surface Tablet).
Paid/Trial
- Grid 3 by SmartBox (2-month trial)
- Communicator 5 by Tobii Dynavox (1-month trial)
Open Source/Free
- OptiKey
- GazeTheWeb
Grid 3
So I downloaded the Grid 3 software by SmartBox (previously owned by Tobii Dynavox who acquired it but had to sell it again) and installed it on the Microsoft Surface.
This was installed without fault and I configured the Tobii eye-tracker pretty easily and a 60 day trial of the software by default.
The UX and UI of this software are nice, from browsing around the web this is distinctly lacking in this category of software.
This product is a mini-suite of applications, each of them with a user type in mind or a function that is needed.
Here are some features (not all):
- Environmental control – turn on your TV, lights, fan, answer the intercom etc if you can have it connected up.
- Computer control – mouse and keyboard emulation.
- Multiple grid-based communication templates for multiple types of users and education levels, such as symbols, words, phrases with images.
- Speech synthesis
- Apps – such as Spotify, Facebook, Whatsapp etc
- Mobile control, text and phone calls, email
- Grid editing and command action buttons
- Eye-tracker calibration in-app
- Web browser
Grid 3 not only looks good but it works well, I have no question why Tobii wanted to keep them.
Communicator 5
I downloaded an installed Communicator 5 by Tobii and it all worked as expected with no obvious issues.
First impressions, it’s teal, I have never been a fan of teal and the overall initial grid that pops up feels dated with tiny fonts on the grid items which don’t seem very well organised.
The whole thing is not as intuitive as the Grid 3 product and I am finding it a little frustrating to use.
But here are some core features from what I found (not all):
- Email, music, calendar, mobile phone connectivity
- Grid creation, modification – which is way too complicated considering what it’s being used for.
- IR connectivity
- Picture grids, learning grids
- Letter Boards
- Calibration with Tobii calibration application, external.
- There is probably more, but I got over it.
All in all the product is probably fit for some circumstances but I didn’t get along with it, I may come back to it as I think for the same price as Grid 3 I’ll stick with that.
OptiKey
This is a free open source Windows solution that comes in 4 parts (applications) depending on the needs of the user.
-
Chat – full-screen keyboard with voice and predictive text
- Symbol – symbol-based communication with voice
- Mouse – mouse simulation
- Pro – all of the others and full computer control
The software was developed by Julius Sweetland and in his words:
“Optikey was written to challenge the outrageously expensive, unreliable and difficult to use AAC (alternative and augmentative communication) products on the market. It is, therefore, fully open-source and free. Forever.”
A great project that I am sure has helped so many disadvantaged people around the world that cannot afford to purchase the alternatives.
OptiKey has most of the options available in the two commercial products I evaluated with the exceptions of mobile, email and specific applications like in Grid 3.
It doesn’t have a convenient way to manipulate the grids like the other products but it is possible, however, this is a little tricky.
There is an interesting plugins component of the product that has:
- Dynamic Keyboards – allows for custom keyboard layouts and actions keys that work with the plugins below.
- External Program Plugin – allows you to trigger external applications with parameters
- HTTP Call Plugin – this will allow you to trigger HTTPS requests for other services, so you could potentially use home automation to trigger lights or change tv channels.
- Custom plugins – you can build your own if you are a developer or have a high degree of tech knowledge.
There is a lot to like about Optikeys suite of tools, the navigation is simple for the most part but I feel it can be a little confusing with the symbol version on some screens.
There is a myriad of control options for setting dwell time etc and it has been well developed.
For my use case, I need a combination of keyboard, plus symbols that are specific to her needs and nothing else confusing or unnecessary. OptiKey Pro does have the ability for all of these options but unfortunately a lot more that may confuse.
The Optikey Symbol is almost what I need, it is a full-screen grid with pause, alert, speech synthesis but the grid would need to be modified to allow only the options necessary. I would also like to be able to trigger a keyboard or letter options for words that need to be spelt out.
I will need to dig more into the grid editing and triggering options to see if I can configure this the way I would like, but it may be possible to use.
This is a free open source solution and may suit some people’s needs.
GazeTheWeb
What I was hoping to find was a browser plug-in or native eye-tracker compatibility built into the common web browsers. That alongside a javascript library and markup for setting the gaze actions would have been awesome, but it doesn’t exist, well not entirely.
Enter GazeTheWeb which is a university project out of Germany specifically for enabling the ‘whole’ web for those that are motor impaired using their eyes (or voice).
SmartBox Grid 3 does have its own web browser and this works pretty well from my testing, but GTW does things a little differently and is not part of a larger suite of tools.
This is not a commercial product and it is a proof of concept prototype, however, it gave me a few ideas that I could test.
I would like to thank Raphael Menges for being patient with me, making some changes to the software and reading my emails.
GazeTheWeb was not intended to work the way I wanted, so I had to make it work with some quirks but I had fun doing it.
I will create a separate post on GTW but I think this little project will have a great future. I am certainly going to keep an eye out for it in the future and hopefully test their commercial product.
See the research paper here on the project.
Using the “Chromium Embedded Framework (CEF)” this is a custom web browser with some interesting features.
The premise of the project is to open up the internet to the motor impaired using eye-gaze technology (and voice) using a dedicated browser rather than mouse emulation whilst keeping all of the core functionality that is available to able-bodied users in the browser.
Digging deeper
This is not a solution out of the box for my use case, however, I found this the most interesting based on what I could potentially use it for.
IF I could get this to work in the way I was thinking then I could in theory build my own custom HTML, JavaScript and APIs and not have to worry about ‘how’ the eye-trackers are integrated.
How it works
Gaze controls for the browser:
- Pause button – pauses the gaze button controls
- Forward, back buttons – previous and next pages
- Tabs – so you can have multiple pages, sites open
- There are extra options with this for bookmarks, URL entry and browsing history.
- Top – takes you to the top of the site, page
- Text selection – allows you to copy sections of text to the clipboard
- Click emulation – click this before you can click links – auto zooms on links
- Auto Scrolling – allows you to scroll by looking toward the bottom or top
- Zoom – page zoom – 1 level
- Mouse emulation – as it says, it emulates the mouse position where your eyes are.
Gaze interactions with the page:
- Page scroll arrows up and down – when page larger than the viewport
- Link – this requires a look at the click emulation control first, then zooms to the link position before emulating the click on the link that is being looked at, if multiple close links.
- Select – this is a gaze point where the select options are on screen and brings up a full-screen list of the options.
- Text – this brings up a gaze controlled keyboard for text area data entry
- The keyboard is a qwerty keyboard with multiple options, word library
I built a little rudimentary web app that I could use GTW to interact with to see how far I could push it (an example in my GTW post that will come later). This worked well, but for the time it would have taken to get it perfect I stopped with it, the solution had already presented itself to me but it was fun.
The final solution
The clear winner is Grid 3 by Smartbox with the top functionality, ease of use and customisation options.
The Whatsapp feature is going to give Mum a way to communicate out of the hospital to people she is not allowed to see when she is ready which is a massive bonus.
I did build a small web app controlled by GazeTheWeb which was a little rudimentary but worked (and free) but this was before I realised the power of Grid 3.
The Microsoft Surface worked well with all software even being an older model and second hand.
I had to configure Windows 10 to auto-login, removed most of the splash screens so there was less friction to boot up and ensured Grid 3 was in the Startup folder so it ran right away. I also configured the power button to go into hibernation mode when pressed, which further accelerated the boot and left fewer potential things to ‘click’ on by the nursing staff when using it.
It is also remotely accessible so I can manage and check on when needed which is a great help, I use Team Viewer for this.
The Tobii 5 gaming eye-tracker (footnote about this) was fantastic, slim-lined and accurate. I am unsure what value you would get out of one that costs 1,000 more considering what this does, but then I haven’t tested it so maybe there is some magic with head position or calibration?
The ZenCT Tablet Stand was great, hardy and tightened and locked into position well on the hospital table.
It appears you can build a solution for under £1,000 which I think is pretty good considering the ‘complete’ options with tablet and software are at least £3k very basic models.
Yesterday (Sept 29th) was our first real success with the setup, she managed to successfully send Whatsapp messages out to a few people as well as receive a few messages all with her eyes.
So all in all I think it has been a successful project so far and I expect Tina (mum) to progress with using it as a tool.
I’ve connected my mother to the internet, untethered her from her bed virtually and given her some of her voice back and that does make me feel a little good.
*Footnote related to Tobii.
I don’t want this to come across as a long diatribe into the Tobii business, I respect the tech but this doesn’t feel right.
On the Smartbox website, I noticed this, (Smart Box used to be owned by Tobii):
“From August 2021, Tobii gaming cameras will no longer be compatible with AAC software including their own titles and Grid 3.
This is because Tobii Dynavox wants to ensure that their eye-tracking solutions used with AAC software can be supported effectively.
This means that any new gaming cameras, such as the recently released Tobii Eye Tracker 5, will not be supported in AAC software – including Grid 3.”
Hmm, considering the gaming variety is 10% of the cost of the AAC version (at the lower end) and it works perfectly well, I have tested it.
I suspect they saw a sudden drop in revenue in the Dynavox division as people who need it most in the AAC market realised it works perfectly well.
You would struggle to sell a £1000+( PCEye 5) to the gamers, they don’t have the critical need for it and this is a market the Tobii tech division has its beady eye on… very lucrative.
I am sure there are a bunch of reasons why the more expensive ones are better etc… but practically, for the price?
Considering Smartbox was part of the Tobii suite of companies and Tobii was made to divest due to competition in the market, perhaps that was a good thing after all.