Google Assistant prepares to proactively show results for “what’s on my screen”

Google Assistant can be summoned by either voice or by long-pressing the home button. But as many of you will recall, before Google Assistant was even a thing, the long-press home action belonged to Google Now on Tap, a feature that scanned what was on your screen and provided contextual information. It was surprisingly accurate and very convenient. The feature made its debut in Google Assistant under a new alias, “What’s on your screen.” But it never became an integral part of Google Assistant and kept appearing and disappearing in various A/B tests. The “What’s on my screen” option used to appear in Google Assistant as a suggestion chip on the main screen of Assistant UI. When you tapped on it, it provided relevant results based on what was displayed on the screen. But it appears the feature could be making a comeback yet again.

An APK teardown can often predict features that may arrive in a future update of an application, but it is possible that any of the features we mention here may not make it in a future release. This is because these features are currently unimplemented in the live build and may be pulled at any time by the developers in a future build.

In our recent APK teardown, we learned that Google is working on bringing a new “My actions” section to the Assistant panel. Now, we’ve discovered that Google is also testing a new “on my screen” section that will appear in this Assistant panel. It doesn’t work at the moment, but we believe it’ll proactively show results that you’d get by previously tapping on the “what’s on my screen” suggestion chip.

This UI test is still very much work in progress, as is evident from the obvious placeholder green color used for the background (likely in preparation for Material You’s new theming system), so it’s not clear if this Assistant UI will roll out with the “on my device” section.

In any case, we would certainly love to see the feature being added to the Assistant — with a reservation that Google doesn’t scrap it once again in one of its A/B tests.

As detailed at Google I/O 2021, Google Assistant is also set to receive a new Suggestion Chips feature that will surface proactive Shortcuts suggestions based on your app usage pattern.

The post Google Assistant prepares to proactively show results for “what’s on my screen” appeared first on xda-developers.