Apple Intelligence integrates context-aware AI across iOS 18, iPadOS 18, and macOS Sequoia

The long-awaited AI integration into Apple's devices arrived just in time for the WWDC24. Apple Intelligence, a privacy-prioritizing, context-aware, personal AI system infused into iOS 18, iPadOS 18, and macOS Sequoia, along the lines of Google's integration of its generative AI models across its apps and Android, or the on-device experience Microsoft has created in the Copilot+ PCs. As Apple describes it, Apple Intelligence "combines the power of generative models with personal context to deliver intelligence that’s incredibly useful and relevant."

One of the most notable features are the new systemwide Writing Tools, which allow users to rewrite, proofread, and summarize text across Mail, Notes, Pages, and third-party apps. Rewrite suggests alternative ways to rephrase with different tones tailored to the audience; Proofread corrects grammar and wording issues with explanations; and Summarize takes selections of text and presents them in a more understandable language, bullet points, a table or a list. Additionally, in Mail users can now profit from Priority Messages a list of urgent messages, like boarding passes or same day invitations. Moreover, rather than previewing the first sentences in an email message, users can now read a summary of the message without opening it. Finally, Smart Reply suggests responses based on the contents of a message, including scanning the message for questions to ensure they are answered in the reply.

Notifications also got an AI-powered revamp. In a similar fashion as the Mail app, urgent notifications are stacked in Priority Notifications, and a summarizing function delivers the most relevant details of long and stacked notifications without having to open them. Moreover, a new Focus mode called Reduce Interruptions blocks all but the most urgent notifications, such as unexpected meetings or changes in scheduling. The summarizing function also works in the Notes and Phone apps when recording and transcribing audio. For instance, when recording a phone call, the participants are automatically notified and at the end of the call, Apple Intelligence delivers a summary of the key points.

Apple Intelligence includes image generation capabilities, as expected. Image Playground enables users to create fun images as animations, illustrations, or sketches. The capability is built into the Messages and Notes apps and is also available as a standalone app. In both cases, the images are created completely on-device and there is no limit to the number of images users can create. Image Playground in Messages is context-aware, which means that it can suggest concepts for image creation related to the participants or topic. Image Playground in Notes is accessible via the new Image Wand feature. This feature can be used to transform sketches into images, or to select blank space to generate an image relevant to the surrounding text. Another image generation feature, Genmoji, allows crafting personalized emojis based on text prompts or someone's photo.

We've seen that generative image capabilities go hand in hand with AI-powered editing, and Apple Intelligence is no different. The Photos app saw significant AI-powered enhancements, like searching using natural language queries and an innovative "Clean Up" tool to remove distracting objects from backgrounds. To leverage the updated Memories feature, users just describe the type of memory they want, and AI automatically curates relevant photos/videos, builds a storyline narrative, and assembles them into a sharable movie.

Unsurprisingly, Siri is the highlight of the newly revamped iOS 18, iPadOS 18, and macOS Sequoia experiences. Siri's enhanced ordinary language capabilities make it more natural, contextually relevant, personal and capable. Siri now accepts voice and text inputs, and has been endowed with enough knowledge to assist users with device support by answering queries on topics such as scheduling messages, or switching from Light to Dark Mode. Being screen-aware makes Siri vastly more useful than previous iterations, as it can learn to act on the content of more apps over time. Cross-app intelligence will empower Siri to handle complex requests spanning multiple apps, such as "Play that podcast that Jamie recommended," or "When is Mom’s flight landing?".

Underpinning these experiences is Apple's innovative Private Cloud Compute system. Most AI-powered features are secure and private because the models powering them are run locally on the device. However, there will always be complex tasks requiring more computing power than a device can provide. In these cases, Private Cloud Compute combines on-device processing with secure routing to larger cloud-hosted models running on Apple silicon, ensuring data is not retained or exposed.

As a complement to Apple Intelligence, Apple is integrating ChatGPT across its platforms, providing access to its knowledge through Siri, Writing Tools and other features, with proper security and privacy guardrails. Users are always asked before any query is redirected to ChatGPT, IP addresses are obscured, and OpenAI does not retain queries coming from Apple Intelligence features. Users without a ChatGPT subscription can use these services free and without a ChatGPT account, and ChatGPT subscribers can connect their accounts to Apple Intelligence and gain access to the paid features directly from the Apple Intelligence experiences.

Overall, the Apple Intelligence features fill an overdue gap, but also feel somewhat muted. Take the Image Playground, for instance. Where other providers have pulled out all the stops to claim that users are able to create anything in any style whatsoever, Apple has decided to restrict some of the more creative image generation features to three styles only. A reasonable explanation emerges almost immediately: Apple frequently bets on delivering consistent and trustworthy experiences, rather than relying on the kind of novelty that feels more experimental, and may work on the short term, but not for much longer than that. With the release of Apple Intelligence, Apple undoubtedly delivered "AI for the rest of us".