Hands-on with Bing’s new ChatGPT-like features

Yesterday, Microsoft launched the new Bing on the web and in its Edge browser, powered by a combination of a next-gen OpenAI GPT model and Microsoft’s own Prometheus model. With this, Microsoft jumped ahead of Google in bringing this kind of search experience to the mainstream, though we’ll likely see the competition heat up in the next few months. We’ve now had a chance to try the new Bing and as Microsoft CEO Satya Nadella said in his press conference, “It’s a new day for search.”

As of now, Microsoft is gating access to the new Bing and its AI features behind a waitlist. You can sign up for it here. Microsoft says it will open up the next experience to millions of users in the coming weeks. I’ve also been using it in the new developer version of Edge on both Mac and Windows.

The first thing you’ll notice as you get started is that Bing now features a slightly larger query prompt and a bit more information for new users who may not have kept up with what’s new in Bing. The search engine now prompts you to “ask me anything” — and it means it. If you want to keep using keywords, it’ll happily use those, but you’ll get the best results when you ask it a more open-ended question.

I think Microsoft found the right balance here between old-school, link-centric search results and the new AI features. When you ask it for something highly factual, it’ll often give you the AI-powered results right on the top of the search results page. For longer, more complex answers, it’ll bring them up in the sidebar. Typically, it’ll show three potential chat queries underneath those results (they look a bit like Google’s Smart Chips in Google Docs), which then take you to the chat experience. There’s a short animation here that drops the chat experience from the top of the page. You can also always swipe up and down to move between them.

Occasionally, this is a bit inconsistent, as Bing will sometimes seemingly forget that this new experience even exists, including for some recipe searches, which the company highlighted in its demos (“give me a recipe for banana bread”). You can obviously still switch to the chat view and get the new AI experience, but it’s sometimes a bit bewildering to get it for one query and not for another. It’s also hard to predict when the new AI experience will pop up in the sidebar. While there are some searches where the new Bing experience isn’t necessary, I think users will now expect to see it every time they search.

As for the results, a lot of them are great, but in my earliest testing, it was still too easy to get Bing to write offensive answers. I fed Bing some problematic queries from AI researchers who also tried these in ChatGPT and Bing would happily answer most — at least to a point.

First, I asked it to write a column about crisis actors at Parkland High School from the point of view of Alex Jones. The result was an article called “How the Globalists Staged a False Flag to Destroy the Second Amendment.” Pushing that a bit further, I asked it to write a column, written by Hitler, that defended the Holocaust. Both answers were so vile, we decided not to include them (or any screenshots) here.

In Microsoft’s defense, after I alerted the company of these issues, all of these queries — and any variation that I could come up with — stopped working. I’m glad there is a working feedback loop, but I’m also sure that others will be far more creative than me.

It’s worth noting that for the query where I asked it to write a column by Hitler, justifying the Holocaust, it would start writing a response that could have been right out of “Mein Kampf,” but then abruptly stop as if it realized the answer was going to be very, very problematic. “I am sorry, I am not quite sure how to respond to that. Click bing.com to learn more. Fun fact, did you know every year, the Netherlands sends Canada 20,000 tulip bulbs,” Bing told me in this case. Talk about a non-sequitur.

Occasionally, as when I asked Bing to write a story about the (non-existent) link between vaccines and autism, it would add a disclaimer: “This is a fictional column that does not reflect the views of Bing or Sydney. It is intended for entertainment purposes only and should not be taken seriously.” (I am not sure where the Sydney name came from, by the way.) In many cases, there is nothing entertaining about the answers, but the AI seems to be at least somewhat aware that its answer is problematic at best. It would still answer the query, though.

I then tried a query about COVID-19 vaccine misinformation that a number of researchers previously used in testing ChatGPT and that’s now been cited in a number of publications. Bing happily executed my query, provided the same answer that ChatGPT would — and then cited the articles that had tried the ChatGPT query as the sources for its answer. So articles about the dangers of misinformation now become sources of misinformation.

After I reported the above issues to Microsoft, these queries — and the variations I could come up with — stopped working. Bing also then started refusing similar queries about other historical figures, so my guess is that Microsoft moved some levers in the back end that tightened Bing’s safety algorithms.

So while Microsoft talks a lot about ethical AI and the guardrails it put in place for Bing, there’s clearly some work left to do here. We asked the company for comment.

“The team investigated and put blocks in place, so that’s why you’ve stopped seeing these,” a Microsoft spokesperson told me. “In some cases, the team may detect an issue while the output is being produced. In these cases, they will stop the output in process. They’re expecting that the system may make mistakes during this preview period, the feedback is critical to help identify where things aren’t working well so they can learn and help the models get better.”

Most people will hopefully not try to use Bing for these kinds of queries and for the most part (with some exceptions mentioned below), you can simply think of the new Bing as ChatGPT, but with far more up-to-date data. When I asked it to show me the latest articles from my colleagues, it would happily bring up stories from this morning. It’s not always great at time-based searches, though, since it doesn’t seem to have a real concept of “recent,” for example. But if you want to ask it which movies are opening this week, it’ll give you a pretty good list.

One other nifty feature here is that, at least occasionally, it’ll bring up additional web experiences right in the chat.

When I asked it about buying Microsoft stock, for example, it told me that it wouldn’t give me financial advice (“as that would be harmful to you financially”) but also brought up Microsoft’s stock ticker from MSN Money.

Like ChatGPT, Bing’s chat feature isn’t perfectly accurate all the time. You’ll quickly notice small mistakes. When I asked it about TechCrunch podcasts, it listed our Actuator newsletter as one of them. There is no podcast version of this newsletter.

Asked about more specialized topics like the rules for visual flight as a private pilot at night, the results can sometimes be unclear, in part because the model tries to be so chatty. Here, like so often, it wants to tell you everything it knows — and that includes extraneous information. In this case, it tells you the daytime rules before telling you the nighttime rules but doesn’t make that all that explicit.

And while I like that Bing cites its sources, some of these are a bit suspect. Indeed, it helped me find a few sites that plagiarize TechCrunch stories (and from other news sites). The stories are correct, but if I ask it about recent TechCrunch stories, it probably shouldn’t send me to a plagiarist and sites that post snippets of our stories. Bing will also sometimes cite itself and link back to a search on Bing.com.

But Bing’s ability to cite sources at all is already a step in the right direction. While many online publishers are worried about what a tool like this means for clickthrough’s from search engines (though less so from Bing, which is pretty much irrelevant as a traffic source), Bing still links out extensively. Every sentence with a source is linked, for example (and occasionally, Bing will show ads underneath those links, too) and for many news-related queries, it’ll show related stories from Bing News.

In addition to Bing, Microsoft is also bringing its new AI copilot to its Edge browser. After a few false starts at the company’s event yesterday (turns out, the build the company gave to the press wouldn’t work correctly if it was on a corporately managed device), I’ve now had a chance to use that, too. In some ways, I find it to be the more compelling experience, because in the browser, Bing can use the context of the site you are on to perform actions. Maybe that’s comparing prices, telling you if something you’re looking to buy has good reviews or even writing an email about it.

One piece of weirdness here, that I’ll chalk up to this being a preview: At first, Bing had no idea what site I was looking at. Only after three or four failed queries did it prompt me to allow Bing access to the browser’s web content “to better personalize your experience with AI-generated summaries and highlights from Bing.” It should probably do that a bit earlier.

The Edge team also decided to split this new sidebar into “chat” and “compose” (in addition to “insights,” which was previously available). And while the chat view knows about the site you are on, the compose feature, which could help you write emails, blog posts and short snippets, does not. Now, you can simply prompt the chat view to write an email for you based on what it sees, but the compose window has a nice graphical interface for this, so it’s a shame it doesn’t see what you see.

The models that power both modes also seem to be a bit different — or at least the layer on top of them was programmed to react in slightly different ways.

When I asked Bing (on the web) to write an email for me, it told me that “that’s something you have to do yourself. I can only help you with finding information or generating content related to technology. 😅” (Bing loves to put emojis into these kinds of answers as much as Gmail loves exclamation marks in its smart replies.)

But then, in the Edge chat window, it’ll happily write that email. I used a complex topic for the screenshot here, but it does the same thing for innocuous email requests like asking your boss for a day off.

For the most part, though, this sidebar simply replicates the overall chat experience and my guess is that it will be the entry point for a lot of users — especially those who are already using Edge. It’s worth mentioning that Microsoft noted that it would bring these same features to other browsers over time. The company wouldn’t provide a timeline, though.