In this post within the "Breaking Down the Biomes" series we're going to take a dive into Siri driven artifacts. Previously we could track some Siri data within KnowledgeC.db but it's now a mixed bag as some of those points have moved out to the biome data. Siri has made it difficult in the past to figure out how it was used as the conversion from the voice data to text data is performed on Apple's end not on device.
With iOS 16 and the movement of a lot of the Siri data, being able to timeline the events are becoming even more important due to Siri's ability to interact with more functions within our devices. Some of the past event such as /siri/ui have been moved out of KnowledgeC.db but it can still be used to track some usage of Siri. In the following screenshot, you can see that there are still events which track back to the voice dictation service being used as well as some SiriIntents being triggered. We'll continue to explore some of these SiriIntents to get a deeper look at what was actually performed by the built-in voice assistant.
Siri.UI
Since the UI triggers are missing from KnowledgeC, we have to turn over to our favorite annoying data store, the biomes! There are several new biomes tied to Siri processes within iOS 16. While this post will cover some of them, there are still more that I noticed within my testing had not yet been populated so more testing will be done over time to see what gets dropped into these.
Starting with the Siri.UI functions, we can use these biomes to track when the UI (user interface) for Siri gets launched. This can help us track when the UI starts and stops to figure out what actions were taken directly following. As a reminder, Apple states that Siri data is processed off your device so what you say to Siri is not going to be left behind. For more information on Siri's processing of data, go to Apple's legal site here.
Looking at the protobufs which reflect "com.apple.siri.ui.begin" the date and time tracks when the UI was triggered for the beginning event. There are two UUIDs listed but the second one is more important (the one in the screenshot ending in 9FC). More on this one as we track through this thread.
The other record reflected in the SEGB file is the type which will reflect "com.apple.siri.ui.end." There are two UUIDs in this record. One for this individual record, and a second UUID which if you note is the same as the previous image. It will also track information about WHY the UI ended. In this case, it says "User" which isn't exactly super helpful, but the UUID can continue to help us.
There is another biome which reflects information relating to the UI in SiriUI found within /private/var/mobile/Library/Biome/streams/public/). The information within the protobufs found inside this SEGB biome file don't tell a lot, unless you're looking at the previous information too. You'll have two events for each Siri activation inside the SEGB, one for the beginning and one for the end.
These date and times are only tracking when these records are written, but the UUID matches the original events from _DKEvent.SiriUI.
The second entry in the pair again shows the UUID, but also has a string value which reflects how the UI was exited. In this case, you can see that the exit reason was "HardwareButton" from the user pressing the button to end. There are other reasons tracked in these strings such as "OutgoingPhoneCall" and "AppLaunch" or "Timeout." Seeing as some of these dismissal reasons mention applications being used, or calls being made, can we track what interaction directly followed the Siri activation?
Post the UI being enabled, we'll shift our attention to what was executed. The biome for SiriExecution (found within /private/var/mobile/Library/streams/public/) will contain a SEGB file with protobufs for execution events in Siri. It should be noted that multiple protobuf records within this file are tied together by a UUID value similar to the UI usage across multiple biomes, however, these UDID records are not tied to the original UI biomes [confused yet? It gets worse].
There are two UUIDs in these protobufs. The first UUID denotes the function such as calls or SMS while the second UUID is tied to the specific execution. The timestamps reflected in each protobuf are signifying when the records are written, not necessarily when the execution occurred.
Several different "intent types" are found that can help show what the execution was tied to such as:
- SendMessageIntent
- LaunchAppIntent
- AppLaunchIntentExtension.LaunchAppIntent
- StartCallIntent
Some intents will also denote what application the intent was tied to such as SendMessageIntent --> com.apple.mobileSMS. Not all will though, some will simply list "none." So how can we use this? Not all executions of the Siri environment have a direct Intent that will leave full traces behind. Some of the intents can be used to help us see what applications may have been launched, some will help us see that some system functions may have been launched.
SiriIntents
While the UI trigger from KnowledgeC.db is gone, SiriIntents are now present in the device. From several scans of images, I was not able to find these prior to iOS 16 images. There are several biomes that relate to SiriIntents but have not been fully populated yet in this version of iOS. Within the KnowledgeC.db the previously mentioned /siri/intentEvent triggers will exist within the ZOBJECT.ZSTREAMNAME column. Taking a look at the previously posted image from KnowledgeC:
SELECT ZOBJECT.ZSTARTDATE AS "START_TIME",
ZOBJECT.ZVALUESTRING AS "VALUE",
ZSTRUCTUREDMETADATA.Z_DKSIRIINTENTEVENTMETADATAKEY__EVENTTYPE AS "TYPE",
ZSTRUCTUREDMETADATA.Z_DKSIRIINTENTEVENTMETADATAKEY__EVENTDATA AS "DATA",
ZOBJECT.ZCREATIONDATE AS "WRITTEN_DATE",
ZOBJECT.ZUUID AS "UUID",
ZOBJECT.Z_PK AS "ZOBJECT_ID"
FROM
ZOBJECT
LEFT JOIN
ZSTRUCTUREDMETADATA
ON ZOBJECT.ZSTRUCTUREDMETADATA = ZSTRUCTUREDMETADATA.Z_PK
WHERE
ZSTREAMNAME = "/siri/intentEvent";
This will return the data similar to this:
The data within Value tracks back to the same intent type as the SiriUI biomes. The BLOB records are... interesting. Buckle up, grab a drink, and let's explore them.
First, examining the BLOB shows that it's a binary Plist file. Inside this plist file there is a record stored with JSON formatting.
Pulling that plist value out and looking at it reveals some information about the intent, but some data stored in the "backingStore" value. These bytes are actually a protobuf.
In the case of one of the recovered CallIntents you can see the information within the protobuf at the end of the line is structured a bit differently, but still reveals to whom the call was placed.
KnowledgeC.db --> BLOB --> Binary Plist --> JSON --> Base64 Encoded Data --> Binary Plist --> Protobuf.
Wrapping Up
What does all of this Siri information get us and how do we use it? What's the TL;DR?
Previously we could track Siri UI event start and stops but almost had to make some assumptions about what might have happened in between. Now we must first look OUTSIDE of KnowledgeC.db into the _DKEvents.SiriUI and SiriUI biomes to figure out when Siri has been triggered (as can see when and how it was exited). In our examples we could see one where some apps were immediately launched and one where a call was started from the exiting of Siri. So how do we track WHAT Siri did if we can? That's where we have to go into our SiriExecution biomes to see if there are any Intents listed or AppLaunches. Remember that Apple tracks AppLaunches in the AppLaunch biome as previously mentioned as well as AppIntents. In addition, by returning back to KnowledgeC.db we can go down the long rabbit hole to return to whom the intent was directed. While this data might not be valuable in every case, I can see several examples where someone might want to know who was messaged or called considering there could be messages or calls deleted which as we have seen does not always get cleared from the biome data storage.
This was a fun, but exhausting little bit of digging to do. I've never seen Apple embed/encode data so deeply before but now at least you can figure out who Siri was calling/messaging. We can also see how Apple are tying the biomes together from the things we have looked at in previous posts. Future posts will continue to track down this data as Apple continues to populate it. Until next time!
No comments:
Post a Comment