Fixes an issue where users were no longer receiving messages after
restoration. During restoration, the restoring device copies the
transferred database files to a special "hotswap" directory since we
don't want to overwrite our currently in-use database. Once finished, it
re-opens the database pointing to our hotswap directory. It would then
move the hotswap database to its primary location on the next app
launch.
The problem here is our extensions don't know how to read the hotswap
directory. Even if we added that capability, it's going to be tricky to
coordinate which database they read and when as the main app shuffles
files around.
This fix adjusts our restoration flow. Instead of having a special
"primary" and "hotswap" directory that we need to fix up on the next
launch, we instead record a UserDefaults entry that points to our
current database directory.
Once transfer has completed, the main app only needs to update the
current database directory in UserDefaults. No post-launch swapping
needs to occur.
Extensions will listen for updates to this database location by
registering KVO on NSUserDefaults.
This change also adjusts the restoration flow to break it up into
discrete stages. The motivation here is we need to make sure that a
partial restoration doesn't put the extensions in an inconsistent state.
Seeing reports of long processing delays in the NSE. We can fetch
messages faster than they can be acked. This backs up our ack queue with
a bunch of repeat work, which means we re-fetch the same envelopes over
and over, which fills the ack queue with more repeat work.
We're seeing crashes during this data migration step. We fail to init
the OWSUserProfile from the Row result returned from the SQLite fetch
request. It's likely we're indexing into an out-of-bounds index on Row.
(Row._checkIndex is inlined so we can't be sure)
GRDB caches the sqlite3_column_count during Row.init and uses that to
determine if a subscript access is out-of-bounds. From what I can tell
in an amateur reading of sqlite3.c, that just pulls the nResColumn from
the prepared statement. SQLite will determine the number of result
columns during statement prep, but the column count can change if the
table schema changes:
> If the database schema changes, instead of returning SQLITE_SCHEMA as
it always used to do, sqlite3_step() will automatically recompile the
SQL statement and try to run it again. As many as
SQLITE_MAX_SCHEMA_RETRY retries will occur before sqlite3_step() gives
up and returns an error.
We're nesting a deferred read transaction inside of our data migration's
immediate transaction. This read is using a different sqlite connection
that the connection performing our migrations. So its understanding of
the schema is stale and because it's in a deferred transaction it won't
figure this out until we call _step().
Once we call _step(), sqlite will realize that the table schema has
changed and recompile the prepared statement while opening the
deferred transaction. After this, the sqlite_column_count is correct.
But Row still has the incorrect value cached from before the schema was
resolved.
So by the time we try and init our OWSUserProfile, Row falsely asserts
that we're indexing beyond the column count by comparing against a stale
version of the column count.
The fix:
Rework the data migration to avoid nesting a deferred read transaction.
This is incorrect behavior. Now that our read is using our already open
write transaction with the same connection handle, it's understanding of
the schema is up-to-date during statement prep. So Row.init grabs the
current column count.
