This adds a table of "cancelled group rings", to handle out-of-order
delivery between a cancellation and a ring. (This can happen when the
cancellation comes from a linked device rather than the original
ringer, though it's very unlikely.) Thirty-minute-old cancellations
are cleared lazily.
This also adds a new IncomingCallControls to replace the usual
CallControls in a GroupCallViewController.
And finally, there's a lot of code that was 1:1-call-specific that now
handles group calls as well, either by being more general or by
checking which kind of call we have.
* Add last read timestamp to StoryContextAssociatedData
* Write lastReadTimestamp on StoryContextAssociatedData
* Copy lastViewedTimestamp over to lastReadTimestamp to simplify badge query
* apply didSets for data that changed at rest
* Add StoryContextAssociatedData and db migrations
* Remove hideStory from ThreadAssociatedData
* Remove lastViewedStoryTimestamp and lastReceivedStoryTimestamp from TSThread
* drop deprecated columns in db migration
* add indexes
* dump schema.sql
* Update unviewed stories SQL query
* fix thread fetching for incoming story messages
* reload story tab badge when StoryContextAssociatedData changes
* Add test for TSGroupModel backwards-compatible deserialization
* move db migration
* Only use StoryContextAssociatedData for story badge count
* update StoryContextAssociatedData lastReceivedTime when a story message is deleted
* catch group threads for outgoing story messages too
* clean up sql query
* Only update lastReceievedStoryTimestamp for remote deletions
* add latestUnexpiredTimestamp to StoryContextAssociatedData
This commit:
1. Adds a new donation receipt type: gifts.
2. Saves receipts when sending someone a gift.
The first part was the main challenge. Previously, donation receipts had
logic like this (pseudocode):
def getReceiptType(self):
if self.subscriptionLevel:
return "subscription"
else:
return "one-time"
Now, we explicitly encode the type and have logic to handle "legacy"
receipts that don't have a type encoded.
Previously, sending a gift badge was not a durable operation, which
meant that crashes/failures could cause users to have their payment
methods charged without actually sending the badge.
Now, the flow is split up into two steps: non-durable parts before the
charge is attempted, and durable parts afterward.
The high-level flow is:
1. Prepare the payment, which involves a couple of repeatable network
requests.
2. Enqueue a job with the prepared payment, that:
1. Charges the payment method (idempotently)
2. Requests a receipt credential (idempotently)
3. Enqueues a gift message, and optionally a text message
3. When the job completes, open the conversation in the UI.
* Little fix for context menu
* Add 'My Stories' section to stories tab
* Add new story thread types
* Show stories in conversation picker
* Support for sending stories
* Update story list when sending stories
* Add basic 'My Stories' view controller
* Initial stories settings screens
* Consolidate TSPrivateStoryThread and TSMyStoryThread into one class
* Require an explicit read transaction to initialize an outgoing message
* Fix linting
* Allow enabling group story from internal settings
* Fix tests
* PR Feedback
This respects the `giftBadges` capability when trying to send gift
badges. In other words, it prevents you from sending gift badges to
someone who lacks the capability.
The bulk of this change involves fetching and saving of this new
capability. The rest of the code involves showing it on the "choose
recipient" screen (and some debug screens).
The existing index was not useful for the query we needed. Running this
query would open a write transaction for several seconds on every
launch. This new index should be much faster.
- Replaces the existing placeholder excluding index for two new indices,
optimized to reduce disk I/O. These are partial indices, only
applicable to the placeholder excluding queries performed during
conversation load. They're also covering indices. Most queries will be
resolved by only consulting the index. There's no need to consult the
backing rows in the model_TSInteraction table.
- Adds a helper SQL bytecode explainer for debugging
Tracking down these optimizations took a great deal of time. I promised
in a comment that this commit message will have a thorough explanation
of why these indices are constructed the way they are. The TL;DR:
- There are a bunch of limitations on index column ordering that you
should be aware of.
- Partial indices can help you avoid some of those limitations
- SQLite can be finnicky when trying to decide if a partial index is
applicable to a query.
- When an index covers a given query, it can avoid a whole bunch of
disk I/O.
- Index row size matters, and SQLite may choose a less efficient index
if it looks like it might be smaller.
The long explanation of lessons learned along the way. All of this is
based on observation, documentation, and trial-and-error:
1. SQLite can be *very* strict when deciding whether or not to use a
partial index. Compare these two index declarations:
> CREATE INDEX index1 ON table(uniqueThreadId, id) WHERE 'recordType IS NOT 70';
> CREATE INDEX index2 ON table(uniqueThreadId, id) WHERE recordType IS NOT 70;
Note the only difference between the two partial index declarations is
wrapping the WHERE clause in quotes. The GRDB index builder we were
using before would do this on our behalf.
Now let's run this query against both indices. The indices we declared
*should* be a perfect choice for this query:
> SELECT COUNT(*) FROM table WHERE uniqueThreadId = 'abc123'
> AND id > 100 AND recordType IS NOT 70;
With the first index:
> `--SEARCH TABLE model_TSInteraction USING INDEX index_interactions_on_threadUniqueId_and_id (uniqueThreadId=? AND id>?)
With the second index:
> `--SEARCH TABLE model_TSInteraction USING INDEX index2 (uniqueThreadId=? AND id>?)
By dropping the quotes around the WHERE clause, SQLite's query planner
is now convinced that the partial index is applicable to the query
we're running. This is good, since as SQLite iterates over the index,
it can skip any recordType comparisons. It also reduces the search
space, since it doesn't even need to read rows it should exclude.
2. Covering indices can be a huge performance win. An index is covering
when every column in the index matches some element. Let's again
compare some indices:
> CREATE INDEX index1 ON table(uniqueThreadId, id) WHERE recordType IS NOT 70;
> CREATE INDEX index2 ON table(uniqueThreadId, id, uniqueId) WHERE recordType IS NOT 70;
> CREATE INDEX index3 ON table(uniqueThreadId, id, recordType, uniqueId) WHERE recordType IS NOT 70;
(Why a difference between index2 and index3? See the next section)
When run on this query:
> SELECT uniqueId FROM table WHERE uniqueThreadId = 'abc123'
> AND id > 100 AND recordType IS NOT 70;
The first index:
> --SEARCH TABLE model_TSInteraction USING INDEX index1 (uniqueThreadId=? AND id>?)
The second index:
> --SEARCH TABLE model_TSInteraction USING INDEX index2 (uniqueThreadId=? AND id>?)
The third index:
> --SEARCH TABLE model_TSInteraction USING COVERING INDEX index3 (uniqueThreadId=? AND id>?)
You can see the difference in the actual query bytecode by running:
"EXPLAIN [query]" (not "EXPLAIN QUERY PLAN"). Notice that when run
against the first index, it needs to perform two reads. One against
the table, and another against the index. The only reason it's
fetching the model is to grab the uniqueId in operation 9.
> 0 Init 0 12 0 0 Start at 12
> 1 OpenRead 0 10 0 67 0 root=10 iDb=0; model_TSInteraction
> 2 OpenRead 1 162160 0 k(3,,,) 0 root=162160 iDb=0; index1
> 3 String8 0 1 0 abc123 0 r[1]='abc123'
> 4 Integer 100 2 0 0 r[2]=100
> 5 SeekGT 1 11 1 2 0 key=r[1..2]
> 6 IdxGT 1 11 1 1 0 key=r[1]
> 7 DeferredSeek 1 0 0 0 Move 0 to 1.rowid if needed
> 8 Column 0 2 3 0 r[3]=model_TSInteraction.uniqueId
> 9 ResultRow 3 1 0 0 output=r[3]
> 10 Next 1 6 0 0
> 11 Halt 0 0 0 0
> 12 Transaction 0 0 198 0 1 usesStmtJournal=0
> 13 Goto 0 1 0 0
3. SQLite will not consider a partial index covering unless the columns
defining the index condition are also included in the index content.
No idea why, but this is why index2 above isn't considered covering
while index3 is.
4. Index column ordering matters. Briefly, if a WHERE clause in a query
has any sort of inequality comparisons (everything but IS, IN and =),
then every subsequent column in an index cannot be used. (mostly)
For example:
> CREATE INDEX ON Table(a,b,c);
> SELECT * WHERE AND b > 100 AND c = "hi";
Column C isn't going to be usable on the index since we're performing
an inequality comparison on B. There are loads of exceptions SQLite
can try to work around this. You can learn more here:
https://www.sqlite.org/optoverview.html
How this applies to this change. Consider these two indices:
> CREATE INDEX index1 ON table(uniqueThreadId, id, recordType, uniqueId) WHERE recordType IS NOT 70;
> CREATE INDEX index2 ON table(uniqueThreadId, recordType, id, uniqueId) WHERE recordType IS NOT 70;
> SELECT uniqueId FROM table WHERE uniqueThreadId = 'abc123'
> AND id > 100 AND recordType IS NOT 70;
index1 is absolutely going to be the better choice here. Since index1
is sorted by id first, it's easy for SQLite to binary search to the
correct offset in the index and then just scan down the index.
(verifying that each recordType IS NOT 70 along the way, which will
always hold true because of the partial index condition).
5. Index size matters. At this point, we've constructed this index:
> CREATE INDEX index1 ON table(uniqueThreadId, id, recordType, uniqueId) WHERE recordType IS NOT 70;
And SQLite is using it for queries like this. Great!
> SELECT uniqueId FROM model_TSInteraction
> WHERE uniqueThreadId = 'whatever' AND recordType IS NOT 70
> ORDER BY id LIMIT 2 OFFSET 0
> `--SEARCH TABLE model_TSInteraction USING COVERING INDEX index1 (uniqueThreadId=?)
Here's a simpler query that we need to run to. It is even less
specific than the one above so we should expect it to use our
index.
> SELECT COUNT(*) FROM model_TSInteraction
> WHERE uniqueThreadId = 'blah' AND recordType IS NOT 70;
> `--SEARCH TABLE model_TSInteraction USING COVERING INDEX index_model_TSInteraction_on_uniqueThreadId_and_hasEnded_and_recordType (uniqueThreadId=?)
As best I can tell, the reason SQLite is picking this existing index
is because each row in the index is smaller.
Each row in our index: [String, Int, Int, String]
Each row in the chosen index: [String, Bool, Int]
The query planner had to choose, was it better off picking the index
that's more specific but larger, or less specific but smaller. It
picked the latter.
To work around this we can add a *second* index, that's even smaller
that the index it's choosing.
> CREATE INDEX index2 ON table(uniqueThreadId, recordType) WHERE recordType IS NOT 70;
Now the query picks our new index to run the count.
--SEARCH TABLE model_TSInteraction USING COVERING INDEX index2 (uniqueThreadId=?)
