Team 06

Team Members

Problem Statement

Everyone has been in a situation where either you or the person you are talking to needs to interrupt the conversation in order to look something up. It breaks the train of thought, causes you to divert attention away from the speaker, and in some cases can be rude. Or maybe you cannot keep up with a lecture, and end up losing valuable intuition about a subject simply because you do not understand a term.

Therefore, we decided to develop a system to mitigate this problem and automate the entire reference searching and note taking process during a conversation. Convo-sense listens to conversation between two peole and does offline analysis to find out relevant context and keywords of the conversation. This preserves user privacy and automates reference searching and note taking for the user.

Specifications

Convo-Sense operates on a Raspberry Pi and uses optimized NLP models to make the user experience flawless. Specifications of the system are:

Capable of detecting conversation from up to 6 feet away.
Audio signal converted to digital text with 95% accuracy from close range and 87% accuracy from 6 feet away.
Captures more than 110 words per minute. Supports four user professions (Engineer, Medical, Financial, and Law).
Displays reference material in less than 6 seconds from the time of speech.
Compatible with any system so long as there is an internet connection and microphone support.

Documentation

=======================Demonstration======================

===================Final Slide Presentation=================

========================FPR Report========================

========================CDR Report========================

========================MDR Report========================

===================PDR Slide Presentation=================

- .... .- -. -.- ....... -.-- --- ..- -.-.--

Sohan Show

I was the Software lead and oversaw the entire system integration and optimization. I primarily worked on the natural language processing that analyzed the conversation and picked out keywords that could be unknown for the user based on their profession. The process included normalizing word frequecy into a json file and writing a custom frequency filter that would keep track of the relevancy of the word according to its last said position in the conversation and the number of occurances. I used SpaCy and NLTK extensively to differentiate relevant words according to two professions and find language context respectively. I also coded up the GUI for the system which showed two different results for two different professions. I used Electron (eel) library to achieve this and coded up the entire GUI in webfront(using HTML/CSS, Javascript). I also setup a Flask server for swift communication between the fron-end and the back-end. Lastly I used the Tkinter library to make a GUI that stores the words that are searched in a button layout which can be clicked to retrieve image result and their definition. In addition to all of this, I oversaw the optimization of the reference script and the speech-to-text script and combined everything into a single functional unit. I also used multi-threading to achieve performance boost and reduced the delay in communication between sub-systems. Lastly, I developed a reset feature which would let the user go back and reselect profession if required.

David Price

I worked primarily on the reference search script. I incorporated openai for definiton search and azure + google image search for the images. The reference search script uses OpenAI’s API to generate an appropriate definition for the keyword chosen based on the context of the sentence that it is said in. We feed the AI the prompt “Please provide the definition of the work {keyword} as used in the following sentence: {sentence}” where {keyword} is the word we want the definition of, and {sentence} is the sentence the system picked up containing the word. Using this system allows us to get the right definitions for words that can have multiple meanings. So, if the user says, “The water current is strong today” and “There is not much current running through the wire”, if current is chosen as the keyword, then the system will get a definition relating to water current for the first sentence and electric current for the second sentence. To get an image, we feed Azure Image Search the keyword and the definition that came back from the AI. We also have a fallback optoin of Google Images search if Azure fails.