A Note-Keeping Application From Google

Google Assistant As a Note Keeping Application

A note-keeping application and other computer applications enable a searcher to create note entries that include content explicitly set by the searcher. For example, the searcher may create a note of “call Bob.” Further, some computer applications may enable limited searching of the searcher-created notes.

This patent refers to Google Assistant many times, and I have written about Google Assistant before. It makes sense to List these previous posts:

• The Google Assistant and Context-Based Natural Language Processing
• Google Automated Assistant Search Results
• How an Automated Assistant May Respond to Queries from Children

The patent looks at Google Assistant as a note-keeping application that people may use to give them reminders and schedule different events daily. I sometimes use paper to write myself notes, use to-do lists on my phone, or send emails to remind myself of something I need to do later. I hadn’t thought of using Google assistant as a note-keeping application, but it does have the ability to help with note-keeping.

Problems with Note keeping Applications

But, The patent tells us that these and other techniques may suffer from drawbacks. For example, a note-keeping application may only enable searching for note entries based on exact keyword matching between terms of a search and terms of the notes. For instance, “call Bob” would only get identified if a search included “call” and “Bob.” Moreover, a note-keeping application may not filter and rank note entries responsive to a search and may only do so based on limited searcher-provided content.

Related Content:

• Technical SEO Agency
• Ecommerce SEO Agency
• Shopify SEO Services
• Franchise SEO Agency
• Enterprise SEO Services

Additionally, a note-keeping application may require that a searcher explicitly choose content provided by the searcher gets utilized to create a note entry and therefore does not create any entries for many types of input of the searcher, such as inputs provided during dialog with a personal assistant and dialog with additional searchers. Additional and alternative drawbacks may get presented.

Natural Language Usage

This specification will generate a personal database entry for a searcher based on free-form natural language input from the searcher via searcher interface input devices of a computer of the searcher, such as natural language input provided to an automated personal assistant and provided to computers of additional searchers during communication with those additional searchers. It is also an effective note-taking application.

The entry includes descriptive metadata based on the natural language input and contextual features associated with receiving it. As used herein, free-form input is input that became formulated by a searcher and is not constrained by options presented for selection by the searcher. For example, it may get typed input provided by the searcher via a physical or virtual keyboard of a computer, spoken input provided by the searcher to a computer microphone, and an image captured by the searcher via a camera. For example, a camera of the computer of the searcher).

Free-Form Natural Language Input of Searchers

This patent is based on a searcher’s personal database entries, output responsive to further free-form natural language input of the searcher. This is part of what makes it a good note-keeping application. Entries that are responsive to the searcher’s natural language input may be identified based on the matching (soft and exact) content of those entries. Such as descriptive metadata and terms of those entries. They would match with search parameters determined based on the further natural language input.

Further, the content of an identified entry may become used in generating the output and ranking multiple identified entries. This would be where multiple entries got identified. They would be relative to one another. The ranking can select a subset of entries (e.g., one entry) to generate the response and determine a presentation order of many pieces of content in a generated response. This would become based on the rankings of the entries from which the many pieces of content got derived.

This specification is additionally and directed to analyzing given natural language input to determine a measure indicative of a likelihood that the input became intended as input the searcher desires an automated personal assistant to recall at a later time and to determine a measure indicative of a likelihood that the input got intended as a request for personal information from a personal database of the searcher. Again, another reason why it is helpful as a note-keeping application. The most effective note-taking application is one that actually takes notes.

A Note-Keeping Application Using Different Natural Language Inputs of Searchers

The natural language input of the searcher may get received and processed in various ways. For example, the searcher may provide natural language input during communication with other searchers via chat, SMS, and other message exchange. Another example is that the natural language input may be provided to an automated personal assistant who engages in a dialog via searcher interface input and output devices. The patent is showing examples of the assistant getting used as a note-keeping application.

For example, the automated personal assistant may become integrated, in whole or in part, in the searcher’s computer. That could be a mobile phone, a tablet, a device devoted solely to the automated assistant functionality. It may include searcher interfaces input devices, such as a microphone or a touch-screen. It may also include searcher interface output devices such as a speaker, a display screen. Also, for example, the automated personal assistant may get implemented, as a note-keeping application, in computers that are separate from, but in communication with, a client computer of the searcher.

Generating Descriptive Metadata For A Note-Keeping Application

How Google Assistant can get used as a note-keeping application. A method performed by processors includes receiving a first natural language input that is free-form input formulated by a searcher via a searcher interface device of a searcher’s computer. The method further involves generating an entry for the first natural language input in a personal database of the searcher stored in computer-readable media. The generating includes: storing, in the entry, given terms of the first natural language input or an identifier of the given terms; generating descriptive metadata based on at least one of:

• The terms of the first natural language input and contextual features associated with receiving the first natural language input
• Storing the descriptive metadata in the entry
• Receiving second natural language input after receiving the first natural language input
• Free-form input formulated by the searcher via the searcher interface input device or an additional searcher interface input device of an additional computer of the searcher
• Determining, based on the second natural language input, at least one search parameter
• Searching the personal database based on the search parameter
• Determining that the entry is responsive to the second natural language input based at least in part on matching the search parameter to at least some of the descriptive metadata
• Generating natural language output that includes natural language output terms that become based on the entry and providing, in response to the second natural language input, the natural language output for presentation to the searcher via a searcher interface output device of the computer or the additional computer

Contextual Features Associated With This Note-Keeping Application Receiving The First Natural Language Input

This method and the technology disclosed may include the following features in some implementations:

• Generating the descriptive metadata get based on both the given terms of the first natural language input and the contextual features associated with receiving the first natural language input.
• Generating the descriptive metadata get based on contextual features associated with receiving the first natural language input
• Generating the descriptive metadata based on the contextual features
• Generating temporal metadata that indicates a date or a time that the first natural language input got received
• Where temporal metadata gets generated, at least one search parameter is a temporal search parameter
• Determining that the entry is responsive to the second natural language input includes matching the temporal search parameter to the temporal metadata
• Where temporal metadata gets generated, the method further includes determining that an additional entry is responsive to the second natural language input and selecting the entry in place of the additional entry based on the conformance of a current date or time to the temporal metadata of the entry
• Providing the natural language output that includes at least some of the given terms of the entry is in response to selecting the entry
• The additional entry is not utilized in generating the natural language output
• No output that becomes based on the additional entry gets provided in response to the second natural language input
• Generating temporal terms of the natural language output based on the temporal metadata of the entry

Generating Descriptive Metadata Based on Contextual Features Associated With Receiving Natural Language Input

Generating the descriptive metadata includes:

• Generating locational metadata that indicates the searcher’s location when the first natural language input gets received
• Where locational metadata gets generated, at least one search parameter is locational
• Determining that the entry is responsive to the second natural language input includes matching the locational search parameter to the locational metadata
• Where locational metadata gets generated, the method includes determining that an additional entry is responsive to the second natural language input
• Selecting the entry in place of the additional entry based on the conformance of a current location of the searcher to the locational metadata of the entry

In some versions of those implementations, providing the natural language output that includes at least some of the given terms of the entry is in response to selecting the entry. The additional entry is not utilized in generating the natural language output. No output based on the additional entry gets provided in response to the second natural language input. In some versions of those implementations, generating the natural language output includes generating locational terms of the natural language output based on the locational metadata of the entry.

Generating Descriptive Metadata Gets Based On Contextual Features

The first natural language input gets provided during communication between the searcher and at least one additional searcher, generating the descriptive metadata gets based on contextual features associated with receiving the first natural language input, and generating the descriptive metadata includes generating additional searcher metadata of the descriptive metadata. The additional searcher metadata identified the searcher in communication when the first natural language input became received.

In some implementations where additional metadata gets generated, the communication between the searcher and the additional searcher is via a first message exchange client of the computer of the searcher and a second message exchange client of an additional searcher computer of the additional searcher:

• Generating descriptive metadata gets based on the first natural language input. Generating the descriptive metadata based on the terms of the first natural language input includes generating a semantic label of the descriptive metadata based on one of the terms
• The semantic label indicates which the terms belong and which additional terms are not included in the first natural language output. In some of those implementations where the semantic label got generated, at least one search parameter is semantic
• Determining that the entry is responsive to the second natural language input includes matching the semantic search parameter to the semantic label. In some versions of those implementations, determining the search parameter got based on a second natural language input prefix

Providing Natural Language Output

In some of those implementations where the semantic label got generated, the method further includes determining that an additional entry is responsive to the second natural language input and selecting the entry in place of the additional entry based on the conformance of the second natural language input of the searcher to the semantic label of the entry. In some versions of those implementations, providing the natural language output that includes at least some of the given terms of the entry is in response to selecting the entry. The additional entry is not utilized in generating the natural language output. No output that got based on the additional entry gets provided in response to the second natural language input.

The method further includes determining, based on at least one of the terms of the second natural language input, a measure indicating the likelihood that the second natural language input is a request for information from the personal database. In some of those implementations, at least one search and the providing depends on the measure’s magnitude.

A method performed by processors gets provided that includes: receiving first natural language input that is free-form input formulated by a searcher via a searcher interface input device of a computer of the searcher and generating an entry for the first natural language input in a personal database of the searcher stored in computer-readable media. The generating includes: storing, in the entry, given terms of the first natural language input or an identifier of the given terms; generating descriptive metadata based on at least one of:
the terms of the first natural language input, contextual features associated with receiving the first natural language input, and storing the descriptive metadata in the entry.

The method further includes:

• Receiving second natural language input after receiving the first natural language input; determining at least one search parameter based on the second natural language input
• Searching the personal database based on the search parameter
• Determining, based on the searching, that the entry is responsive to the second natural language input
• Determining, based on the searching, that an additional entry is also responsive to the second natural language input
• Ranking the entry relative to the additional entry based on at least some of the descriptive metadata
• Generating natural language output that includes natural language output terms that got based on the entry
• Providing, in response to the second natural language input, the natural language output for presentation to the searcher via a searcher interface output device
• Wherein providing the natural language output got based on the ranking of the entry relative to the additional entry

Additional Features of The Note-Keeping Application Patent

This method and other implementations of the technology disclosed herein may each optionally include the following features:

• Providing the natural language output based on the ranking of the entry relative to the additional entry includes providing the natural language output without providing any output based on the additional entry in response to the second natural language input
• Providing the natural language output based on the ranking of the entry relative to the additional entry includes providing the natural language output and providing additional output based on the additional entry, and providing the natural language output and the additional output for presentation in an order that got based on the ranking
• In addition, some implementations include processors of computers, where the processors are operable to execute instructions stored in associated memory, and where the instructions got configured to cause the performance of any of the aforementioned methods. Some implementations also include a non-transitory computer-readable storage medium storing computer instructions executable by processors to perform any aforementioned methods
• It should become appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein got contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure gets considered part of the subject matter disclosed herein

The note-keeping application patent is at:

Generating a personal database entry for a searcher based on natural language searcher interface input of the searcher and generating output based on the entry in response to further natural language searcher interface input of the searcher
Inventors: Maryam Garrett, Wan Fen Nicole Quah, Bryan Horling, and Ruijie He
Assignee: GOOGLE LLC
US Patent: 10,783,178
Granted: September 22, 2020
Filed: May 17, 2016

Abstract

Some implementations get directed to generating a personal database entry for a searcher based on free-form natural language input formulated by the searcher via searcher interface input devices of a computer of the searcher. The generated personal database entry may include terms of the natural language input and descriptive metadata determined based on the natural language input and/or based on contextual features associated with receiving the natural language input. Some implementations get directed to generating, based on personal database entries of a searcher, output that is responsive to further free-form natural language input of the searcher. For example, entries that are responsive to further natural language input of the searcher can become identified based on matching the content of those entries to search parameters determined based on the further input.

Implementations of this specification got directed to generating a personal database entry for a searcher based on free-form natural language input formulated by the searcher via searcher interface input devices of a computer of the searcher. For example, the personal database entry may become generated by an automated personal assistant in response to natural language input provided to the automated personal assistant.

A personal database entry, as used herein, is an entry in a personal database that is accessible to the searcher and non-accessible to a plurality of additional searchers that differ from the searcher. For example, the personal database may be accessible to only the searcher and applications and systems specified by the searcher and non-accessible to any other searchers, applications, and systems.

Also, for example, the personal database may be accessible to the searcher and certain additional searchers designated by the searcher. As described herein, any personal database may become stored on computer-readable media and may get associated with access information to allow access only to searchers, applications, and systems authorized to access such content and prevent access to all other searchers, applications, and systems.

A generated personal database entry includes terms of the natural language input. It includes descriptive metadata based on the natural language input and contextual features associated with receiving it.

Descriptive metadata determined based on the terms of the natural language input may include, for example, a semantic label indicating a classification to which of the terms belong and to which additional terms not included in the natural language output also belong. For example, a semantic label of “person” may get determined based on the term “Bob,” a semantic label of “place” may become determined based on the term “home,” etc.

Another example of descriptive metadata determined based on the terms of the natural language input may include “memoryless” metadata that indicates the likelihood (e.g., true/false, or a range of values) that the natural language input got intended as input the searcher desires an automated personal assistant to be able to recall at a later time.

For example, descriptive memoryless metadata that indicates a high likelihood the searcher desires the ability for future recall may become generated based on the presence of certain key terms in the input (e.g., “remember,” “don’t forget”) and based on output received in response to providing the input to a classifier trained to predict whether and to what degree natural language input indicates a desire for the natural language input to be “remembered.”

Descriptive Metadata Based on Contextual Features From Natural Language Input

The descriptive metadata determined based on the contextual features associated with receiving the natural language input may include, for example:

• Temporal metadata
• Locational metadata
• Additional searcher metadata

The temporal metadata may indicate a date (e.g., May 1, 2016; May; 2016; last week; Sunday) and a time (e.g., 8:00 AM, morning, between 7:00 and 9:00) that the natural language input received. The locational metadata may indicate the location of the searcher (e.g., a ZIP code, “home,” “work,” a City, a Neighborhood) when the natural language input got received.

The extra searcher metadata may indicate additional searchers to whom the natural language input got directed. Additional searchers engaged in communication with the searcher when the natural language input became received. For example, when the natural language input becomes received from an ongoing message exchange thread (e.g., chat, SMS) between the searcher and an additional searcher, the additional searcher metadata may indicate the additional searcher.

Implementations of this specification are additionally and directed to generating, based on personal database entries of a searcher, output that is responsive to further free-form natural language input of the searcher. In some of those implementations, entries that are responsive to further natural language input of the searcher can get identified based on matching (soft and exact) content of those entries (e.g., descriptive metadata and natural language input terms of those entries) to search parameters determined based on the further natural language input.

Further, in some of those implementations content of an identified entry may become utilized in generating the output and ranking multiple identified entries (where multiple entries got identified) relative to one another. The ranking can select a subset of entries to generate the response and determine a presentation order of multiple pieces of content in a generated response (based on the rankings of the entries from which the multiple pieces of content got derived).

An example environment in which techniques disclosed herein get implemented and illustrated

The example environment includes a plurality of client computers and an automated personal assistant. However, the automated personal assistant is separate from the client computers. All aspects of the automated personal assistant got implemented by the client computers. For example, a client computer may implement one instance or more automated personal assistants and the client computer. N may also implement a separate instance of those aspects of an automated personal assistant. In implementations where aspects of automated personal assistant get implemented by computers remote from client computers, the client computers and those aspects of automated personal assistant may communicate via networks such as a local area network (LAN) and wide area network (WAN) (e.g., the Internet).

The client computers may include, for example, a desktop computer, a laptop computer, a tablet computer, a dedicated computer that enables dialog with searchers (e.g., a stand-alone device that includes a microphone and speaker but no display), a mobile phone computer, a computer of a vehicle of the searcher (e.g., an in-vehicle communications system, an in-vehicle entertainment system, an in-vehicle navigation system), or a wearable apparatus of the searcher that includes a computer (e.g., a watch of the searcher having a computer, glasses of the searcher having a computer, a virtual or augmented reality computer).

Additional and alternative client computers may get provided. A given searcher may communicate with an automated personal assistant utilizing a plurality of client computers of the searcher that collectively form a coordinated “ecosystem” of computers. For example, a personal database entry may get generated based on the natural language input via a first client computer and output generated based on that entry in response to further natural language input of the searcher via a second client computer. However, for the sake of brevity, many examples described in this disclosure will focus on a given searcher operating a single one of the client computers.

Each client computer may operate various applications, such as a corresponding one of the message exchange clients. Message exchange clients may come in various forms and vary across the client computers. sub.1-N and multiple forms may operate on a single one of the client computers. In some implementations of the message exchange, clients may come in the form of a short messaging service (“SMS”) and multimedia messaging service (“MMS”) client, an online chat client (e.g., instant messenger, Internet relay chat, or “IRC,” etc.), a messaging application associated with a social network, a personal assistant messaging service dedicated to dialog with an automated personal assistant (e.g., implemented on a dedicated computer that enables dialog with a searcher), and so forth. In some implementations of the message exchange, clients may get implemented via a web page or other resources rendered by a web browser (not depicted) or other client computer application.

As described in more detail herein, the automated personal assistant receives input from and provides output to searchers via searcher interface input and output devices of client devices.sub.1-N. Implementations described herein relate to the automated personal assistant generating personal database entries for a searcher based on natural language input provided by the searcher and generating output for presentation to the searcher based on generated personal database entries of the searcher.

However, it becomes understood that the automated personal assistant may also perform additional functions in many implementations. For example, the automated personal assistant may respond to a searcher’s natural language interface input with output that gets generated, in whole or in part, based on information that is not generated based on a personal database entry generated based on past natural language input of the searcher. For example, the automated personal assistant may utilize documents of a public database in generating a response to some natural language interface input of the searcher. For example, in response to searcher natural language input of “what does ambidextrous mean?”, the automated personal assistant may generate an output of “ambidextrous is an adjective that describes the ability to use the right and left hands equally well” based on a definition for “ambidextrous” that is not personal to the searcher.

The searcher interface input described herein is explicitly directed to the automated personal assistant. For example, one message exchange client may be a personal assistant messaging service dedicated to conversations with an automated personal assistant. Searcher interface input provided via that personal assistant messaging service may be automatically provided to an automated personal assistant. For instance, one of the message exchange clients and an instance of at least aspects of the personal assistant may both get implemented at least in part on a stand-alone client computer that at least selectively monitors for voice searcher interface input and responds with audible searcher interface output.

Also, for example, the searcher interface input may be explicitly directed to the automated personal assistant via the message exchange clients based on particular searcher interface input that indicates the automated personal assistant is to get invoked. For instance, the particular searcher interface input uses typed characters (e.g., @PersonalAssistant), searcher interaction with a virtual button (e.g., a tap, a long tap), a verbal command (e.g., “Hey Personal Assistant”), etc. The automated personal assistant may respond to searcher interface input, even when that searcher interface input is not explicitly directed to the automated personal assistant. For example, the automated personal assistant may examine the contents of searcher interface input and perform actions in response to certain terms present in the searcher interface input and based on other cues.

Each of the client computers and automated personal assistant may include memories for storage of data and software applications, processors for accessing and executing applications, and other components that facilitate communication over a network—the client computers. sub.1-N and by the automated personal assistant may get distributed across multiple computer systems. Automated personal assistants may get implemented as, for example, computer programs running on computers in locations that get coupled to each other through a network.

Automated personal assistants may include an input processing engine, a contextual engine, an entry generation engine, a searching engine, a ranking engine, and an output generation engine. Engines may become omitted. Aspects of engines may get combined. Engines may get implemented in a separate component from an automated personal assistant. For example, engines, or any operative part thereof, may get implemented in a component executed by the client computers.

Input processing engine processes natural language input generated by searchers via client computers. sub.1-N generates annotated output for use by other components of the automated personal assistant. For example, the input processing engine may process natural language free-form input generated by a searcher via searcher interface input devices of the client device. The generated annotated output includes annotations of the natural language input and optionally (e.g., all) of the terms of the natural language input.

The input processing engine gets configured to identify and annotate various types of grammatical information in natural language input. For example, the input processing engine may include a speech tagger configured to annotate terms with their grammatical roles. For example, the part of speech tagger may tag each term with its part of speech such as “noun,” “verb,” “adjective,” “pronoun,” etc. The input processing engine may include a dependency parser configured to determine syntactic relationships between terms in natural language input. For example, the dependency parser may determine which terms modify other terms, subjects, and verbs of sentences, and so forth (e.g., a parse tree)–and may make annotations of such dependencies.

The input processing engine may additionally include an entity tagger configured to annotate entity references in segments such as references to people, organizations, locations, etc. The entity tagger may annotate references to an entity at a high level of granularity (e.g., to enable identification of all references to an entity class such as people) and a lower level of granularity (e.g., to enable identification of all references to a particular entity such as a particular person). The entity tagger may rely on the content of the natural language input to resolve a particular entity. It may optionally communicate with a knowledge graph or other entity database to resolve a particular entity.

The input processing engine may include a coreference resolver configured to the group, or “cluster,” referencing the same entity based on contextual cues. For example, the coreference resolver may get utilized to resolve the term “it” to “bike helmet” in the natural language input “I’m looking for my bike helmet. Do you know where it is?”

The input processing engine components may rely on annotations from other components of the input processing engine. The named entity tagger may rely on annotations from the coreference resolver and dependency parser in annotating all mentions to a particular entity. The coreference resolver may rely on annotations from the dependency parser in clustering references to the same entity.

In processing a particular natural language input, components of the input processing engine may use related prior input and other related data outside the particular natural language input to determine annotations. For example, a first searcher in a message exchange thread may provide input of “I’ll leave the keys under the doormat for you,” and a second searcher may provide, via the message exchange thread, input to the automated messaging system of “Remember that.” In processing “Remember that,” the coreference resolver may resolve “that” to “I’ll leave the keys under the doormat for you” and may further resolve “I’ll” to the first searcher and “you” to the second searcher.

The contextual engine determines contextual features associated with natural language input received from one of the client computers. The contextual engine determines the contextual features independent of the natural language input. For example, contextual features may indicate a date and a time that the natural language input got received and may get independently determined based on the natural language input. Also, for example, a contextual feature may indicate a location of the searcher (e.g., based on GPS or other location data) when the natural language input got received and may get determined independently of terms of the natural language input.

Also, for example, a contextual feature may indicate additional searchers to whom the natural language input got directed and additional searchers who became engaged in communication with the searcher when the natural language input got received–and may get determined independently of terms of the natural language input. The contextual engine determines the contextual features independent of any explicit input of the searcher that provided the natural language input, such as explicit searcher interface input that defines the contextual features and explicit input that confirms automatically determined contextual features are correct.

The entry generation engine generates personal database entries for a searcher in response to some natural language input formulated by the searcher via a corresponding one of the client computers. In generating an entry for received natural language input, the entry generation engine may utilize terms of the natural language input, annotations of the natural language input provided by the input processing engine, and contextual features provided by the contextual engine.

As one example, assume natural language input of “remember my locker combination is 10, 20, 30” formulated by a searcher via one of the client computers and provided to the automated personal assistant. Further, assume the contextual engine determines the natural language input provided at 6:00 AM on May 1, 2016, while the searcher’s location is “at the gym.” The searcher’s location of “at the gym” may become defined with any of various levels of granularity, such as a specific latitude/longitude, a street address, an entity identifier that identifies the particular gym, an entity class identifier of “gym,” etc.

In such an example, the entry generation engine may generate an entry that includes the terms “locker combination is 10, 20, 30”; temporal metadata of 6:00 AM, “early morning,” and some other indication of 6:00 AM and temporal metadata of “May 1.sup.st”, “Sunday in May,” and some other identifier of “May 1, 2016”; and locational metadata that is an indication of the searcher’s location of “at the gym.”

The entry generation engine may generate a personal database entry based on received natural language input in response to that natural language input having characteristics. For example, in some of those implementations, the entry generation engine may generate a personal database entry in response to natural language input that includes key terms (e.g., “remember,” “remember this,” and “don’t forget”) and that includes those key terms in certain positions of the natural language input (e.g., as a prefix).

Also, for example, in some of those implementations, the entry generation engine may additionally and generate a personal database entry in response to providing at least some of natural language input to a classifier trained to predict whether and to what degree natural language input indicates a desire for the natural language input to be “remembered”–and receiving, as output from the classifier, an indication that the natural language input indicates a desire to become remembered and indicates the desire to at least a threshold degree.

The entry generation engine may also generate “memoryless” metadata for an entry that indicates a likelihood that the natural language input utilized to generate the entry indicates a desire for the automated personal assistant to recall at least some of the natural language input later. For example, the entry generation engine may generate memoryless metadata that is the first value (e.g., “1”) when the natural language input includes key terms (e.g., “remember,” “remember this,” and “don’t forget”) and is a second value (e.g., “0”) when the natural language input does not include any of those key terms.

Also, for example, the entry generation engine may generate memoryless metadata selected from one of three or more values based on the natural language input characteristics and associated contextual features. For example, the entry generation engine may provide at least some of the received natural language input and contextual features to a classifier trained to predict whether and to what degree natural language input indicates a desire for the natural language input to be “remembered”–and receive, as output from the classifier, a measure of the likelihood that the natural language input indicates a desire for the natural language input to become remembered. The engine may utilize that measure, or another indication based on that measure, as the memoryless metadata.

The personal database entries generated by the entry generation engine may become stored in the personal database. A personal database may become provided on non-transitory computer-readable media, such as local media, to one client’s computer. sub.1-N, local to the automated personal assistant and remote from the client computers. sub.1-N and the automated personal assistant. Each personal database entry may include the underlying content itself and index items that link to the underlying content. The personal database may be local to one of the client’s computers. That client computer may implement an instance of (e.g., all) aspects of the automated personal assistant. For example, that client computer may include searcher interface input devices (e.g., a microphone), searcher interface output devices (e.g., a speaker), and the personal database.

A personal database enables searching of personal database entries to determine whether all or aspects of an entry are relevant to further natural language input of the searcher as described herein. Multiple iterations of the personal database may get provided, with each iteration being specific to a particular searcher. They may include personal database entries accessible to the searcher and system(s) designated by the searcher and non-accessible to a plurality of additional searchers and systems that differ from the searcher.

For example, the personal database may be accessible only to the searcher and non-accessible to any other searcher. Also, for example, the personal database may be accessible to the searcher and certain additional searchers designated by the searcher. The personal database may become associated with access information to allow searchers authorized to access such content and prevent access to all other searchers. The personal database includes entries of a plurality of searchers, with each entry and a group of entries associated with access information to allow access only to searchers and systems authorized to access such index items and prevent access to all other searchers and systems.

Thus, the personal database may include entries for a plurality of searchers. Still, each entry and a group of entries may include access information to prevent searchers not authorized to access such entries. Additional and alternative techniques for restricting access to the database personal database entries may have gotten used.

The searching engine searches entries of a personal database associated with a searcher in response to at least some natural language input formulated by the searcher via a corresponding one of the client computers. The searching engine generates search parameters based on the formulated natural language input, and optionally based on annotations of that input provided by the input processing engine, and utilizes those search parameters to determine entries of the personal database that include content (e.g., descriptive metadata and natural language input terms of those entries) that matches (soft match and exact match) (e.g., all) of the search parameters.

A search parameter generated by the searching engine may include, for example, terms of the natural language input, a synonym of terms of the natural language input, a semantic label for terms of the natural language input, a location restriction based on terms of the natural language input, a temporal restriction based on terms of the natural language input, etc. For example, for natural language input of “who did I give my bike helmet to last month?” the searching engine may determine search parameters of “bike helmet” (based on terms of the natural language input), “bicycle helmet” (based on a synonym of the natural language input), a semantic label indicative of “person” (based on presence of “who” in the natural language input), and a temporal restriction corresponding to “last month.”

The searching engine searches the personal database based on received natural language input in response to that natural language input having characteristics. For example, in some of those implementations, the searching engine may search the personal database in response to natural language input that includes key terms (e.g., “What,” “Who,” “Where,” “When,” “my,” and “I”) and that includes those key terms in certain positions of the natural language input. For instance, the search engine may search the personal database in response to natural language input that includes an inquiry term such as “what,” “who,” or “where” with a syntactical relationship (e.g., as indicated by annotations of the input processing engine) to terms such as “I” or “my.”

The searching engine may additionally and search the personal database in response to providing at least some of natural language input to a classifier trained to predict whether and to what degree natural language input became intended as a request for personal information from a personal database of the searcher–and receiving, as output from the classifier, an indication that the natural language input indicates a desire and indicates a desire to at least a threshold degree.

The searching engine may also search the public database in response to some natural language input formulated by the searcher via a corresponding one of the client computers. The searching engine may search both the public dad personal databases response to some natural language input and may search only one of the public databases and the personal databases; generally, the public database may include various types of content that are not restricted to the searcher and other searcher’s and systems designated by the searcher, and that may get utilized by the searching engine to respond to various types of natural language input.

For example, the public database may include content items from publicly accessible documents on the World Wide Web, scripted responses to certain natural language input, etc. For example, the public database may include a definition for “ambidextrous” to enable the searching engine to identify that definition in response to natural language input of “what does ambidextrous mean?”. As another example, the public database may include a scripted response of “Hello, how are you” that got identified by the searching engine in response to natural language input of “Hi” or “Hello.”

The ranking engine optionally ranks entries (personal and public) identified by the searching engine based on criteria. The ranking engine optionally calculates a score for each of the entries responsive to the query using the criteria. The score of an entry may itself get used as the ranking for the entry or may get compared to scores of other entries (where other entries got determined) to determine the ranking of the entry. Each criterion used to rank an entry provides information about the entry itself and its relationship with the natural language input.

The ranking of a given entry may become based on the conformance of a semantic label of descriptive metadata of the entry to a semantic label generated for the natural language input. For example, if a semantic label for terms of the natural language input matches a semantic label in descriptive metadata of the given entry, the ranking of the given entry may be positively influenced.

The ranking of the given entry may get based on the conformance of the locational metadata of the given entry to the searcher’s current location of.g., determined by the contextual engine) when the natural language input got received. For example, if a current location is the same as, or within a threshold distance of, a location indicated by locational metadata of the given entry, the ranking of the given entry may be positively influenced.

The ranking of the given entry may additionally and got based on the conformance of temporal metadata of the given entry to the current date and time (e.g., determined by a contextual engine) when the natural language input got received. For example, if the current date is within a threshold of a date indicated by temporal data of the given entry, the given entry may get promoted more than if the current date was not within the threshold.

The ranking of the given entry may additionally and got based on memoryless metadata for the given entry. For example, if a measure included in or defined by the memoryless metadata satisfies a threshold, the ranking of the given entry may have gotten promoted more than if the measure did not meet the threshold. Where the searching engine identifies a first entry from the public database and a second entry from the personal database, the rankings of the entries may become based on whether and to what degree the natural language input got intended as a request for personal information from a personal database of the searcher (e.g., as described above concerning the searching engine).

For example, if characteristics of the natural language input indicate it is highly likely to be a request for personal information from a personal database of the searcher, any identified entries from the public database may get demoted.

The output generation engine utilizes the content of entries identified by the searching engine and optionally the rankings of those entries determined by the ranking engine to generate output for presentation to the searcher as a response to the provided natural language input. For example, the output generation engine may select a subset (e.g., one) of identified entries based on the ranking and generate output based on the selected subset.

Also, for example, the output generation engine may additionally utilize the ranking to determine a presentation order of multiple pieces of content in a generated response (based on the rankings of the entries from which the multiple pieces of content got derived).

In generating output based on a given personal database entry, the output generation engine may include e terms of the given entry to the natural language input for which the given entry got generated and may include output based on generated descriptive metadata of the entry.

The patent shows additional descriptions of various components of automated personal assistants.

searcher Interface Input Devices to Provide Natural Language Input to Message Exchange Client Which Transmits the Natural Language Input to Input Processing Engine

The natural language input may be free-from input as described herein and may be, for example, typed input provided by the searcher via a physical or virtual keyboard of a computer or spoken input provided by the searcher to a microphone of the computer. In implementations where the natural language input got spoken input, it may get transcribed to textual input by the computer and the input processing engine.

The input processing engine processes the natural language input and generates various annotations of the natural language input. The input processing engine provides annotated input (e.g., terms of the natural language input and generated annotations) to the entry generation engine.

The contextual engine determines contextual features associated with receiving the natural language input. For example, the contextual engine may determine the contextual features based on the message exchange client (e.g., a timestamp, locational data) and other applications and the client computer’s operating system. The contextual features may become utilized by the entry generation engine described below. It may be by the input processing engine (e.g., to improve entity resolution and other determinations based on searcher location or other contextual features).

The entry generation engine generates a personal database entry for a searcher and stores it in a personal database. In generating the entry, the entry generation engine may utilize terms of the natural language input, annotated input of the natural language input provided by the input processing engine, and contextual features 204 provided by the contextual engine.

Illustrates an example of generating, based on personal database entries of a searcher, natural language output that is responsive to further free-form natural language input of the searcher. The example of FIG. 2B may occur after and may use the entry.

A searcher utilizes searcher interface input devices of the computer.sub.1 to provide further natural language input to message exchange client.sub.1, which transmits the natural language input to the input processing engine. The natural language input may be free-from input as described herein and may be, for example, typed input provided by the searcher via a physical or virtual keyboard of a computer or spoken input provided by the searcher to a microphone of the computer.

The input processing engine processes the natural language input and generates various annotations of the natural language input. The input processing engine provides annotated output (e.g., terms of the natural language input and generated annotations) to the search engine.

The searching engine generates search parameters based on the annotated input. It utilizes those search parameters to determine personal and public databases entries that match the search parameters. The searching engine may search only personal databases in response to the natural language input based on the natural language input characteristics. The searching engine determines that entry of personal database is responsive to the search based on matching search parameters to the entry content.

The searching engine provides the entry to the ranking engine, which generates a ranking for the entry. For example, the ranking engine may determine if a ranking score for the entry satisfies a threshold that warrants the output generation engine utilizing the entry to generate natural language output. The contextual engine determines contextual features associated with receiving the natural language input and may provide those contextual features to the ranking engine for use in generating a ranking as described herein. The ranking engine determines many responsive entries. The ranking engine may rank those relative to one another.

The output generation engine utilizes the entry content to generate natural language output for presentation to the searcher to respond to the natural language input. For example, the output generation engine may generate output that includes e terms of the entry to the natural language input and includes terms based on generated descriptive metadata for the entry. The natural language output provided to the message exchange client for audible and visual presentation to the client computer’s searcher interface output device.

The searching engine identifies multiple responsive entries, and the output generation engine may select a subset (e.g., one) of identified entries based on the ranking. It may generate output based on the selected subset. Also, for example, the output generation engine may additionally utilize the ranking to determine a presentation order of multiple pieces of content in a generated response (based on the rankings of the entries from which the many pieces of content got derived).

Additional description gets provided of various components and techniques

The implementations described herein illustrate the client computer with a display screen displaying an example of dialog between a searcher of the client computer and the automated personal assistant. In particular, the dialog includes natural language input provided by the searcher to an automated personal assistant and responses C that may optionally get provided by the automated personal assistant to confirm the assistant has generated an entry in response to the inputs.

In some other implementations, an alternative response may get provided by the automated personal assistant. Where clarification of a natural language input may be necessary and beneficial in determining a more particularized entry, the automated personal assistant may respond with a prompt to seek such further clarification. For instance, in response to natural language input, the automated personal assistant may identify both a “Bob Smith” and “Bob White” in contact entries of the searcher and may generate a prompt such as “Do you mean Bob Smith or Bob White?” to solicit further input from the searcher to enable determination of a particular contact and further tailoring of an entry based on the input.

Also, for instance, in response to natural language input, the automated personal assistant may determine that the input conflicts with previous input and the assistant 120 might prompt “Should this replace the bike lock combination you provided on May 5, 2016?” to determine if an entry generated based on the input should replace an entry generated based on the input.

FIG. 3A also shows, in parentheses, a date and a time that the searcher provided each of the various inputs 380A-D. As described herein, the dates and times may get utilized in generating descriptive metadata. The dates and times may optionally not become shown in the display provided to the searcher but get illustrated for clarity.

The display screen further includes a textual reply interface element that the searcher may select to generate searcher input via a virtual keyboard and a voice reply interface element 385 that the searcher may select to generate searcher input via a microphone. The display screen also includes system interface elements that the searcher may interact with cause the computer to perform actions.

Figures each illustrate a portion of the example client computer, with the display screen displaying a dialog that may occur after the dialog according to implementations described herein.

Later, the searcher provides further natural language input of “who has my bike helmet?”.

The automated personal assistant responds, “On May 1.sup.st you told me that Bob had your bike helmet.” The response may get generated by: issuing a search based on “where is my bike helmet”; identifying, as responsive to the search, an entry that got created based on the searcher input; and using that entry to generate the response. For example, an entry generated for searcher input may include the terms “Bob has my bike helmet” and descriptive metadata of a semantic label of “person” associated with “Bob,” a semantic label of “object” associated with “bike helmet,” and temporal metadata indicating May 1.sup.st and 8:00 AM.

Also, for example, a search may get issued based on the searcher input, such as a search that includes a first search parameter of “bike helmet” and a second search parameter of a semantic label of “person” (based on the presence of “who” in the searcher input). The input entry may get identified based on terms matching the first search parameter and a semantic label matching the second. The response may get generated to include temporal metadata from the entry for input (“On May 1.sup.st”) and the text from the entry for input (“Bob has your Bike helmet”).

ThAt later, the searcher provides natural language input of “when did I put tires on my bike?”. The automated personal assistant responds of “May 7, 2016” The response may get generated by: issuing a search based on “when did I put tiers on my bike”; identifying, as responsive to the search, an entry that got created based on the searcher input; and using that entry to generate the response 390F. For example, an entry generated for searcher input may include “put new tires on my bike on May 7, 2016”. The entry may include the terms “May 7, 2016” instead of “today” (which got provided in the searcher input) based on the entry generation engine replacing “today” with the date the searcher input got received (e.g., based on annotations provided by input processing engine). The entry generated for searcher input may also include descriptive metadata of temporal metadata indicating May 7, 2016, and a semantic label of “when” associated with the entry’s terms “May 7, 2016”.

Also, for example, a search may become issued based on the search input, such as a search that includes a first search parameter of “tires,” a second search parameter of “bike,” and a third search parameter of a semantic label of “when” (based on presence of “when” in the searcher input). The entry generated for input may become identified based on having terms that match the first search parameter and the second search parameter and having semantic label metadata that matches the third search parameter. The response 390F may become generated to include only the terms “May 7, 2016” from the entry for input based on those terms associated with the semantic label metadata that matches the third search parameter.

The drawings illustrate further natural language inputs of “What is my bike lock combination.” A first entry generated for searcher input and a second for searcher input may get identified based on a search for natural language inputs. A response that got generated based on the second entry for searcher input got provided without providing any output that gets generated based on the first entry for searcher input. This may get released on ranking the first and second based on criteria and determining the second entry gets ranked higher than the first. For example, the second entry may get ranked higher based on temporal metadata, for the second entry is more proximal to a current time than the first one.

The response got generated based on the second entry for searcher input without providing any output based on the first entry. Providing the content from only one of the two answers may get based on various factors such as system settings, display screen size, and input processing to understand that only a single answer gets desired. A response may include output generated based on the first and the second entries.
An illustration got generated based on the second entry for searcher input and the first entry for searcher input. The response that corresponds to the second entry gets presented before the portion based on the first entry. This may get based on, for example, the ranking of the second entry being higher than the ranking of the first entry (e.g., based on temporal metadata for the second entry being more proximal to a current time than temporal metadata for the first entry).

ThaAt a later time, the searcher provides the natural language input of “How do I put tires on my bike?”. The automated personal assistant provides a response that includes part of the steps for putting new tires on a bike. The response is an example of a response generated by the automated personal assistant based on a public entry of a public database. Although the further natural language input includes the terms “tires” and “bikes” that may also get included in a personal entry generated based on the searcher input, the automated personal assistant does not provide responsive content that becomes based on the personal entry.

This may get based on many various factors. For example, a semantic label for the input 380H (e.g., a label associated with “How”) may not match a semantic label for the personal entry generated based on the searcher input. This may lead to that entry not becoming determined and may negatively influence the ranking of that entry.

As another example, the ranking for the public entry utilized to generate response may be higher than the ranking of the personal entry–and the response generated based on only the public entry based on a rule that dictates responses are to get generated based on only the highest-ranked entry and based on only the highest-ranked entry when its ranking score satisfies a threshold relative to the next highest-ranked entry.

Later, the searcher provides further natural language input of “new tires on my bike?”. The automated personal assistant provides a response 390J containing content from a private and public database. In particular, the response 390J includes content (“You put new tires on your bike on May 7, 2016”) that gets based on a personal entry generated for the input (FIG. 3A). The response also includes content (“Do you want to learn how to install new bike tires?”) that may get based on a public entry of the public database.

The automated personal assistant may include content that gets based on an entry generated for the input based on matching search parameters of “new,” “tires,” and “bike” to the content of the entry. The content from the private database entry may get presented before the content from the public database based on a ranking of the entries underlying the content.

The Display Screen Showing Dialog Between the Searcher And The Automated Personal Assistant

In particular, the dialog includes natural language inputs provided by the searcher to the automated personal assistant and responses that may optionally get provided by the automated personal assistant to confirm the assistant has generated an entry in response to the inputs. An illustration also shows, in parentheses, a date and a time that the searcher provided each of the inputs. As described herein, the dates and times may get utilized in generating descriptive metadata. The dates and times may optionally not get shown in the display provided to the searcher.

The display screen further includes a textual reply interface element that the searcher may select to generate searcher input via a virtual keyboard and a voice reply interface element that the searcher may select to generate searcher input via a microphone. The display screen also includes system interface elements that may get interacted with by the searcher to cause the computer.sub.1 to perform actions.

The searcher provides first searcher input that includes “my locker combination is 7, 22, 40” and provides it at the “Gym” on “May 1, 2016” at “6:00 AM”. A first entry may get created for the first searcher input that includes terms from the input, includes locational metadata indicative of the “Gym,” and includes temporal metadata indicative of May 1, 2016, and 6:00 AM. The searcher provides second searcher input that includes “my locker combination is 40, 20, 10” and provides it at “Work” on “May 10, 2016” at “8:00 AM”. A second entry may get created for the first searcher input that includes terms from the input, includes locational metadata indicative of “Work,” and includes temporal metadata indicative of May 10, 2016, and 8:00 AM.

Figures each illustrate a part of the example client computer with the display screen displaying a dialog that may occur subsequently.

The searcher provides further input of “What is my locker combination?” and provides it at 5:45 AM while the searcher is at the Gym. The automated personal assistant provides a response generated based on the first entry for the searcher input–and that does not include any content generated based on the second entry for the searcher input. This may become based on, for example, conformance of the locational metadata of the first entry to the current location of the searcher and conformance of the temporal metadata to a current time.

Compare this to a situation where the searcher was at “Work” at “8:00 AM”. In such a situation t, the response may get generated based on the second end independent of the first.

The searcher provides further input of “What is my locker combination?” The searcher provides the input at 6:00 PM while the searcher is at “Home.” The automated personal assistant provides the response generated based on both the first entry for the searcher. Generating a response based on both entries instead of just one may get based on, for example, the searcher’s current location not conforming strongly to the locational metadata of either of the entries and the current time not conforming strongly to the temporal metadata of either of the entries.

In response, the content generated based on the second entry may get presented before content generated based on the first entry based on the date of temporal metadata of the second entry being more proximal to the current date than that of the first entry.

We may see example graphical searcher interfaces that may get provided to a searcher. Still, it gets understood that responses provided by an automated personal assistant may get provided for presentation audibly (e.g., via a speaker), without necessarily providing graphical indications of those responses. For example, a searcher may provide natural language input via a microphone, and the automated personal assistant may respond to such input with output configured for presentation audibly.

Generating Note-Keeping Application entries For a Searcher Based on Natural Language Input Responsive to Free-Form Input of the Searcher

This system may include various components of various computer systems, such as automated personal assistants. Moreover, while methods are in a particular order, this is not meant to be limiting. The Process From this patent describes a note-keeping application.

This patented process receives natural language input that gets formulated by a searcher via searcher interface input devices.

The system determines whether the natural language input indicates a request for personal information, such as a request for personal information from a personal database of the searcher. For example, the system may determine the input indicates a request for personal information if it includes key terms.

Those terms could look like “What,” “Who,” “Where,” “may,” “I,” and “don’t forget.” That includes key terms in certain positions of input. Also, for example, the system may determine input indicates a request for personal information based on output received in response to providing at least some input to a classifier trained to predict whether and to what degree natural language input becomes intended as a request for personal information.

It determines the natural language input does not state a request for information, the system proceeds to determine whether the input indicates a desire for recall. For example, the system may determine the input indicates a desire for recall based on certain terms in the input. Terms such as “remember” “don’t forget.”

Those could also get based on output received in response to providing the input to a classifier trained to predict whether and to what degree natural language input indicates a desire for the natural language input to be “remembered.” Processes get performed for each input or for each input determined not to state a request for personal information, which satisfies other criteria.

If the process determines, the input does not state a desire for recall, the system proceeds and awaits further natural language output after providing an error output. That might sound like “I don’t understand your request.” The next steps might involve other tasks such as searching a public database and providing other output.

Suppose it is determined that the natural language input does state a desire for recall, the system proceeds where the system generates an entry for the input in a personal database of the searcher. This is where it acts as a note-keeping application.

We may see the generation of descriptive metadata. That could get based on input and contextual features associated with receiving the input. It will then store input and generate descriptive metadata in an entry. The system may then perform other actions, such as providing output that confirms an entry generated for the input, then proceeds back.

It can determine that the input does state a request, the system proceeds.

The process may also search a personal database of the searcher based on the search parameters.

It could look to determine at least one personal entry of the personal database that is responsive to the search. The system generates natural language output based on the content of at least one person entry. The system may generate output based on a personal entry only when a ranking for that personal entry satisfies a threshold. The ranking may be for the personal entry or a personal entry relative to other personal entries that may be determined to be responsive to the search.

It may provide the presentation to the searcher via searcher interface output devices. The system then returns and awaits further natural language input.

The patented process may search public databases and may generate output with content from the public databases. Such public database content may get included in output in combination with content from a personal entry and place of content from the personal entry.

About Bill Slawski

With more than 26 years of SEO experience and a Juris Doctor Degree, Bill Slawski is the foremost expert on Google’s patents as related to SEO. Patent Exploration is one of the quickest and most detailed ways to find new information about SEO. Bill is the Editor of SEO by the Sea, a prominent search engine optimization blog, where he is the author of over 1,300 posts. Bill’s experience includes Fortune 500 brands and some of the largest websites in the world. Bill is a contributing author for Moz, Search Engine Land, and Search Engine Journal. In 2014-2021, he spoke at industry-leading international conferences about topics including search engine algorithms, universal and blended search, personalization in search, search and social, and duplicate content problems, structured data, and schema