Barak Turovsky Analyzes AI’s Natural Language Processing Revolution

This article originally appeared on InformationWeek.

As more and more low-code platforms arise, the acceleration of IT automation being adopted in the enterprise continues to grow. Generative AI and its ability to impact our lives has been one of the hottest topics in technology, especially regarding ChatGPT.

There is immense promise but potentially even greater risks with this approach to automation. Will we see IT automation in enterprises entering a new phase of adoption?

In this archived keynote session, Barak Turovsky, VP of AI at Cisco, reveals the maturation of AI and computer vision and its impact on the natural language processing revolution. This segment was part of our live virtual event titled, “Strategies for Maximizing IT Automation.” The event was presented by ITPro Today and InformationWeek on March 28, 2024.

A transcript of the video follows below. Minor edits have been made for clarity.

Barak Turovsky: So, AI for speech recognition, as I mentioned, was this era that now has subsided in terms of understanding what the system is. The reason it became so popular is because several companies invested in the technology, and it finally matured.

You could use deep neural networks to get a very high degree of confidence in speech recognition. The second benefit of AI is that it's basically computer vision, which again, is unstructured data where you can recognize a dog or use a device to translate anything visually.

Related:Cisco VP of AI Explains LLM Use Cases and IT Automation Benefits

Any sign or any menu item can be translated quickly and even be used in augmented reality. I’m very proud of all those early innovations that we made on one of my teams at Google Translate. Machine learning as a tool at that time was what we now call AI, because Google was an early adopter of the technology.

During that period, we used statistical machine learning for natural language processing. To some extent in the past, I had to explain why natural language processing was so important because in many cases, the first wave of the AI revolution was limited to companies like Google.

Google by design is a language company, but with the power of ChatGPT today, we know how important language processing is. On a higher level, the technology industry wants to enable users to manage their world with the power of language.

Whether you type or talk, this is the most natural interface, and language processing is a critical component of many technology products. Today, I don't think I need to explain language processing, but in the past, I did because it was limited to companies like Google.

It's an extremely complex topic. That's why the breakthrough with this technology is relatively recent. Ultimately, it allows the industry to achieve higher levels of natural language processing capabilities. It’s very complex because languages are hard, and these are real world examples.

Languages can be ambiguous. For example, a meaningful learning goal could be in a programming language or game, or it could vary, since there are multiple ways to say the same thing. I could say I worked for Google, or I could say that Google employed me.

Language also involves some slang. It also could be extremely contextual, and humans understand that, but machines won’t until very recently. The number of variances in language is insane, which is why machines require different technologies to better understand the nuances.

In languages with AI, the breakthrough was utilizing neural networks on huge amounts of training data. With natural language processing for one language, you’re able to better understand what someone said in English, and I will show you a couple of examples.

The second breakthrough was neural machine translation. This is what I call bilingual, natural language processing, where you understand language A, and translate it to language B. Both served as the baseline for this evolution.

To give you a little bit of history, the generative evolution started in 2017. As I mentioned, in 2016, my team at Google launched neural machine translation for what we now call commercial LLMs. It was a huge technological breakthrough that we’ll talk about, both on a software and hardware level.

This led to a new wave of research that culminated in a paper known as Transformer, Attention is All You Need. This was basically the breakthrough that enabled the current generative AI revolution because it showed new ways of processing data, and especially understanding what people say to generate responses.

The first product was known as a bidirectional encoder, which is a product that allowed us to look at both directions of text. We could also do pre-training and fine tuning. That was the first productization of transformative technology in 2018 that was initially done for Google search, which then expanded to many other products at Google.

It was eventually open sourced by Google. So obviously, Google deserves a ton of credit for what they contributed to the industry that sparked this revolution. I feel extremely lucky and proud that I was part of that amazing journey.

So, from a high level, what Bidirectional Encoder Representations from Transformers (BERT) does is hide roughly 20% of the world as we train and retrain. We're asking the neural model to try to guess those words, and to some extent predict the right word to use.

In this case for example, words at the top like grass, habitats, called, ground, mammals, and small are basically hidden. To just guess new words is not necessarily that useful, but if you train the model on an insane amount of data from billions of training prompts, it starts to become very good at trying to create question answering framework.

So, for example, if the question is what are mammals that build nests on the ground? You give some context that most mammals like squirrels build nests high in the trees, while raccoons may build theirs on the ground.

This allows the model to predict the right answers, and that’s a super simplistic use of BERT.

Watch the archived “Strategies for Maximizing IT Automation” live virtual event on-demand today.