- Replaces the existing placeholder excluding index for two new indices,
optimized to reduce disk I/O. These are partial indices, only
applicable to the placeholder excluding queries performed during
conversation load. They're also covering indices. Most queries will be
resolved by only consulting the index. There's no need to consult the
backing rows in the model_TSInteraction table.
- Adds a helper SQL bytecode explainer for debugging
Tracking down these optimizations took a great deal of time. I promised
in a comment that this commit message will have a thorough explanation
of why these indices are constructed the way they are. The TL;DR:
- There are a bunch of limitations on index column ordering that you
should be aware of.
- Partial indices can help you avoid some of those limitations
- SQLite can be finnicky when trying to decide if a partial index is
applicable to a query.
- When an index covers a given query, it can avoid a whole bunch of
disk I/O.
- Index row size matters, and SQLite may choose a less efficient index
if it looks like it might be smaller.
The long explanation of lessons learned along the way. All of this is
based on observation, documentation, and trial-and-error:
1. SQLite can be *very* strict when deciding whether or not to use a
partial index. Compare these two index declarations:
> CREATE INDEX index1 ON table(uniqueThreadId, id) WHERE 'recordType IS NOT 70';
> CREATE INDEX index2 ON table(uniqueThreadId, id) WHERE recordType IS NOT 70;
Note the only difference between the two partial index declarations is
wrapping the WHERE clause in quotes. The GRDB index builder we were
using before would do this on our behalf.
Now let's run this query against both indices. The indices we declared
*should* be a perfect choice for this query:
> SELECT COUNT(*) FROM table WHERE uniqueThreadId = 'abc123'
> AND id > 100 AND recordType IS NOT 70;
With the first index:
> `--SEARCH TABLE model_TSInteraction USING INDEX index_interactions_on_threadUniqueId_and_id (uniqueThreadId=? AND id>?)
With the second index:
> `--SEARCH TABLE model_TSInteraction USING INDEX index2 (uniqueThreadId=? AND id>?)
By dropping the quotes around the WHERE clause, SQLite's query planner
is now convinced that the partial index is applicable to the query
we're running. This is good, since as SQLite iterates over the index,
it can skip any recordType comparisons. It also reduces the search
space, since it doesn't even need to read rows it should exclude.
2. Covering indices can be a huge performance win. An index is covering
when every column in the index matches some element. Let's again
compare some indices:
> CREATE INDEX index1 ON table(uniqueThreadId, id) WHERE recordType IS NOT 70;
> CREATE INDEX index2 ON table(uniqueThreadId, id, uniqueId) WHERE recordType IS NOT 70;
> CREATE INDEX index3 ON table(uniqueThreadId, id, recordType, uniqueId) WHERE recordType IS NOT 70;
(Why a difference between index2 and index3? See the next section)
When run on this query:
> SELECT uniqueId FROM table WHERE uniqueThreadId = 'abc123'
> AND id > 100 AND recordType IS NOT 70;
The first index:
> --SEARCH TABLE model_TSInteraction USING INDEX index1 (uniqueThreadId=? AND id>?)
The second index:
> --SEARCH TABLE model_TSInteraction USING INDEX index2 (uniqueThreadId=? AND id>?)
The third index:
> --SEARCH TABLE model_TSInteraction USING COVERING INDEX index3 (uniqueThreadId=? AND id>?)
You can see the difference in the actual query bytecode by running:
"EXPLAIN [query]" (not "EXPLAIN QUERY PLAN"). Notice that when run
against the first index, it needs to perform two reads. One against
the table, and another against the index. The only reason it's
fetching the model is to grab the uniqueId in operation 9.
> 0 Init 0 12 0 0 Start at 12
> 1 OpenRead 0 10 0 67 0 root=10 iDb=0; model_TSInteraction
> 2 OpenRead 1 162160 0 k(3,,,) 0 root=162160 iDb=0; index1
> 3 String8 0 1 0 abc123 0 r[1]='abc123'
> 4 Integer 100 2 0 0 r[2]=100
> 5 SeekGT 1 11 1 2 0 key=r[1..2]
> 6 IdxGT 1 11 1 1 0 key=r[1]
> 7 DeferredSeek 1 0 0 0 Move 0 to 1.rowid if needed
> 8 Column 0 2 3 0 r[3]=model_TSInteraction.uniqueId
> 9 ResultRow 3 1 0 0 output=r[3]
> 10 Next 1 6 0 0
> 11 Halt 0 0 0 0
> 12 Transaction 0 0 198 0 1 usesStmtJournal=0
> 13 Goto 0 1 0 0
3. SQLite will not consider a partial index covering unless the columns
defining the index condition are also included in the index content.
No idea why, but this is why index2 above isn't considered covering
while index3 is.
4. Index column ordering matters. Briefly, if a WHERE clause in a query
has any sort of inequality comparisons (everything but IS, IN and =),
then every subsequent column in an index cannot be used. (mostly)
For example:
> CREATE INDEX ON Table(a,b,c);
> SELECT * WHERE AND b > 100 AND c = "hi";
Column C isn't going to be usable on the index since we're performing
an inequality comparison on B. There are loads of exceptions SQLite
can try to work around this. You can learn more here:
https://www.sqlite.org/optoverview.html
How this applies to this change. Consider these two indices:
> CREATE INDEX index1 ON table(uniqueThreadId, id, recordType, uniqueId) WHERE recordType IS NOT 70;
> CREATE INDEX index2 ON table(uniqueThreadId, recordType, id, uniqueId) WHERE recordType IS NOT 70;
> SELECT uniqueId FROM table WHERE uniqueThreadId = 'abc123'
> AND id > 100 AND recordType IS NOT 70;
index1 is absolutely going to be the better choice here. Since index1
is sorted by id first, it's easy for SQLite to binary search to the
correct offset in the index and then just scan down the index.
(verifying that each recordType IS NOT 70 along the way, which will
always hold true because of the partial index condition).
5. Index size matters. At this point, we've constructed this index:
> CREATE INDEX index1 ON table(uniqueThreadId, id, recordType, uniqueId) WHERE recordType IS NOT 70;
And SQLite is using it for queries like this. Great!
> SELECT uniqueId FROM model_TSInteraction
> WHERE uniqueThreadId = 'whatever' AND recordType IS NOT 70
> ORDER BY id LIMIT 2 OFFSET 0
> `--SEARCH TABLE model_TSInteraction USING COVERING INDEX index1 (uniqueThreadId=?)
Here's a simpler query that we need to run to. It is even less
specific than the one above so we should expect it to use our
index.
> SELECT COUNT(*) FROM model_TSInteraction
> WHERE uniqueThreadId = 'blah' AND recordType IS NOT 70;
> `--SEARCH TABLE model_TSInteraction USING COVERING INDEX index_model_TSInteraction_on_uniqueThreadId_and_hasEnded_and_recordType (uniqueThreadId=?)
As best I can tell, the reason SQLite is picking this existing index
is because each row in the index is smaller.
Each row in our index: [String, Int, Int, String]
Each row in the chosen index: [String, Bool, Int]
The query planner had to choose, was it better off picking the index
that's more specific but larger, or less specific but smaller. It
picked the latter.
To work around this we can add a *second* index, that's even smaller
that the index it's choosing.
> CREATE INDEX index2 ON table(uniqueThreadId, recordType) WHERE recordType IS NOT 70;
Now the query picks our new index to run the count.
--SEARCH TABLE model_TSInteraction USING COVERING INDEX index2 (uniqueThreadId=?)
- Annotate groupIds fetched from the UD messgae content as untrusted
- Double check the sender of a failed decryption is a member of the
group before inserting a placeholder/error message
- Double check group membership hasn't changed before deciding not to
expire a sender key.
