Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
Comment: | Reactivate query flattening when the result set of the outer query has no function calls or subqueries. This is a partial reversal of check-in [c9104b59]. Co-routines are still preferred if the outer query has a complex result set, but for simple results sets, query flattening is used. Check-in [4464f40ccd7] is completely backed out due to this change. |
---|---|
Downloads: | Tarball | ZIP archive |
Timelines: | family | ancestors | descendants | both | trunk |
Files: | files | file ages | folders |
SHA3-256: |
d17ef7d153058f7332b3fec421ade42c |
User & Date: | drh 2017-10-28 20:51:54.404 |
Context
2017-10-28
| ||
20:54 | Increase the version number for the next release - which is still months away but there have been significant query planner enhancements since the previous release. (check-in: 457eedfac0 user: drh tags: trunk) | |
20:51 | Reactivate query flattening when the result set of the outer query has no function calls or subqueries. This is a partial reversal of check-in [c9104b59]. Co-routines are still preferred if the outer query has a complex result set, but for simple results sets, query flattening is used. Check-in [4464f40ccd7] is completely backed out due to this change. (check-in: d17ef7d153 user: drh tags: trunk) | |
12:20 | Add test cases from OSSFuzz to prevent a regression in co-routine processing. (check-in: 689743d8e3 user: drh tags: trunk) | |
2017-10-04
| ||
12:06 | Turn restriction 20 on the query flattener into an assert since the situation restricted can no longer occur because of the more aggressive use of co-routines. (check-in: 4464f40ccd user: drh tags: trunk) | |
Changes
Changes to src/expr.c.
︙ | ︙ | |||
948 949 950 951 952 953 954 955 956 957 958 959 960 961 | assert( pToken ); pNew = sqlite3ExprAlloc(db, TK_FUNCTION, pToken, 1); if( pNew==0 ){ sqlite3ExprListDelete(db, pList); /* Avoid memory leak when malloc fails */ return 0; } pNew->x.pList = pList; assert( !ExprHasProperty(pNew, EP_xIsSelect) ); sqlite3ExprSetHeightAndFlags(pParse, pNew); return pNew; } /* ** Assign a variable number to an expression that encodes a wildcard | > | 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 | assert( pToken ); pNew = sqlite3ExprAlloc(db, TK_FUNCTION, pToken, 1); if( pNew==0 ){ sqlite3ExprListDelete(db, pList); /* Avoid memory leak when malloc fails */ return 0; } pNew->x.pList = pList; ExprSetProperty(pNew, EP_HasFunc); assert( !ExprHasProperty(pNew, EP_xIsSelect) ); sqlite3ExprSetHeightAndFlags(pParse, pNew); return pNew; } /* ** Assign a variable number to an expression that encodes a wildcard |
︙ | ︙ |
Changes to src/select.c.
︙ | ︙ | |||
3379 3380 3381 3382 3383 3384 3385 | ** (18) If the sub-query is a compound select, then all terms of the ** ORDER BY clause of the parent must be simple references to ** columns of the sub-query. ** ** (19) If the subquery uses LIMIT then the outer query may not ** have a WHERE clause. ** | | | | | | < | 3379 3380 3381 3382 3383 3384 3385 3386 3387 3388 3389 3390 3391 3392 3393 3394 3395 3396 3397 | ** (18) If the sub-query is a compound select, then all terms of the ** ORDER BY clause of the parent must be simple references to ** columns of the sub-query. ** ** (19) If the subquery uses LIMIT then the outer query may not ** have a WHERE clause. ** ** (20) If the sub-query is a compound select, then it must not use ** an ORDER BY clause. Ticket #3773. We could relax this constraint ** somewhat by saying that the terms of the ORDER BY clause must ** appear as unmodified result columns in the outer query. But we ** have other optimizations in mind to deal with that case. ** ** (21) If the subquery uses LIMIT then the outer query may not be ** DISTINCT. (See ticket [752e1646fc]). ** ** (22) The subquery may not be a recursive CTE. ** ** (**) Subsumed into restriction (17d3). Was: If the outer query is |
︙ | ︙ | |||
3518 3519 3520 3521 3522 3523 3524 3525 3526 3527 3528 3529 3530 3531 | /* Restriction (17): If the sub-query is a compound SELECT, then it must ** use only the UNION ALL operator. And none of the simple select queries ** that make up the compound SELECT are allowed to be aggregate or distinct ** queries. */ if( pSub->pPrior ){ if( isAgg || (p->selFlags & SF_Distinct)!=0 || pSrc->nSrc!=1 ){ return 0; /* (17d1), (17d2), or (17d3) */ } for(pSub1=pSub; pSub1; pSub1=pSub1->pPrior){ testcase( (pSub1->selFlags & (SF_Distinct|SF_Aggregate))==SF_Distinct ); testcase( (pSub1->selFlags & (SF_Distinct|SF_Aggregate))==SF_Aggregate ); assert( pSub->pSrc!=0 ); | > > > | 3517 3518 3519 3520 3521 3522 3523 3524 3525 3526 3527 3528 3529 3530 3531 3532 3533 | /* Restriction (17): If the sub-query is a compound SELECT, then it must ** use only the UNION ALL operator. And none of the simple select queries ** that make up the compound SELECT are allowed to be aggregate or distinct ** queries. */ if( pSub->pPrior ){ if( pSub->pOrderBy ){ return 0; /* Restriction (20) */ } if( isAgg || (p->selFlags & SF_Distinct)!=0 || pSrc->nSrc!=1 ){ return 0; /* (17d1), (17d2), or (17d3) */ } for(pSub1=pSub; pSub1; pSub1=pSub1->pPrior){ testcase( (pSub1->selFlags & (SF_Distinct|SF_Aggregate))==SF_Distinct ); testcase( (pSub1->selFlags & (SF_Distinct|SF_Aggregate))==SF_Aggregate ); assert( pSub->pSrc!=0 ); |
︙ | ︙ | |||
3552 3553 3554 3555 3556 3557 3558 | ** The only way that the recursive part of a CTE can contain a compound ** subquery is for the subquery to be one term of a join. But if the ** subquery is a join, then the flattening has already been stopped by ** restriction (17d3) */ assert( (p->selFlags & SF_Recursive)==0 || pSub->pPrior==0 ); | < < < < < < < < < | 3554 3555 3556 3557 3558 3559 3560 3561 3562 3563 3564 3565 3566 3567 | ** The only way that the recursive part of a CTE can contain a compound ** subquery is for the subquery to be one term of a join. But if the ** subquery is a join, then the flattening has already been stopped by ** restriction (17d3) */ assert( (p->selFlags & SF_Recursive)==0 || pSub->pPrior==0 ); /***** If we reach this point, flattening is permitted. *****/ SELECTTRACE(1,pParse,p,("flatten %s.%p from term %d\n", pSub->zSelName, pSub, iFrom)); /* Authorize the subquery */ pParse->zAuthContext = pSubitem->zName; TESTONLY(i =) sqlite3AuthCheck(pParse, SQLITE_SELECT, 0, 0, 0); |
︙ | ︙ | |||
4339 4340 4341 4342 4343 4344 4345 4346 4347 4348 4349 4350 4351 4352 | int i, j, k; SrcList *pTabList; ExprList *pEList; struct SrcList_item *pFrom; sqlite3 *db = pParse->db; Expr *pE, *pRight, *pExpr; u16 selFlags = p->selFlags; p->selFlags |= SF_Expanded; if( db->mallocFailed ){ return WRC_Abort; } if( NEVER(p->pSrc==0) || (selFlags & SF_Expanded)!=0 ){ return WRC_Prune; | > | 4332 4333 4334 4335 4336 4337 4338 4339 4340 4341 4342 4343 4344 4345 4346 | int i, j, k; SrcList *pTabList; ExprList *pEList; struct SrcList_item *pFrom; sqlite3 *db = pParse->db; Expr *pE, *pRight, *pExpr; u16 selFlags = p->selFlags; u32 elistFlags = 0; p->selFlags |= SF_Expanded; if( db->mallocFailed ){ return WRC_Abort; } if( NEVER(p->pSrc==0) || (selFlags & SF_Expanded)!=0 ){ return WRC_Prune; |
︙ | ︙ | |||
4451 4452 4453 4454 4455 4456 4457 4458 4459 4460 4461 4462 4463 4464 4465 4466 4467 4468 4469 4470 4471 4472 4473 4474 4475 4476 4477 4478 4479 | */ for(k=0; k<pEList->nExpr; k++){ pE = pEList->a[k].pExpr; if( pE->op==TK_ASTERISK ) break; assert( pE->op!=TK_DOT || pE->pRight!=0 ); assert( pE->op!=TK_DOT || (pE->pLeft!=0 && pE->pLeft->op==TK_ID) ); if( pE->op==TK_DOT && pE->pRight->op==TK_ASTERISK ) break; } if( k<pEList->nExpr ){ /* ** If we get here it means the result set contains one or more "*" ** operators that need to be expanded. Loop through each expression ** in the result set and expand them one by one. */ struct ExprList_item *a = pEList->a; ExprList *pNew = 0; int flags = pParse->db->flags; int longNames = (flags & SQLITE_FullColNames)!=0 && (flags & SQLITE_ShortColNames)==0; for(k=0; k<pEList->nExpr; k++){ pE = a[k].pExpr; pRight = pE->pRight; assert( pE->op!=TK_DOT || pRight!=0 ); if( pE->op!=TK_ASTERISK && (pE->op!=TK_DOT || pRight->op!=TK_ASTERISK) ){ /* This particular expression does not need to be expanded. */ | > > | 4445 4446 4447 4448 4449 4450 4451 4452 4453 4454 4455 4456 4457 4458 4459 4460 4461 4462 4463 4464 4465 4466 4467 4468 4469 4470 4471 4472 4473 4474 4475 | */ for(k=0; k<pEList->nExpr; k++){ pE = pEList->a[k].pExpr; if( pE->op==TK_ASTERISK ) break; assert( pE->op!=TK_DOT || pE->pRight!=0 ); assert( pE->op!=TK_DOT || (pE->pLeft!=0 && pE->pLeft->op==TK_ID) ); if( pE->op==TK_DOT && pE->pRight->op==TK_ASTERISK ) break; elistFlags |= pE->flags; } if( k<pEList->nExpr ){ /* ** If we get here it means the result set contains one or more "*" ** operators that need to be expanded. Loop through each expression ** in the result set and expand them one by one. */ struct ExprList_item *a = pEList->a; ExprList *pNew = 0; int flags = pParse->db->flags; int longNames = (flags & SQLITE_FullColNames)!=0 && (flags & SQLITE_ShortColNames)==0; for(k=0; k<pEList->nExpr; k++){ pE = a[k].pExpr; elistFlags |= pE->flags; pRight = pE->pRight; assert( pE->op!=TK_DOT || pRight!=0 ); if( pE->op!=TK_ASTERISK && (pE->op!=TK_DOT || pRight->op!=TK_ASTERISK) ){ /* This particular expression does not need to be expanded. */ |
︙ | ︙ | |||
4595 4596 4597 4598 4599 4600 4601 | } } } } sqlite3ExprListDelete(db, pEList); p->pEList = pNew; } | > | | | > > > > | 4591 4592 4593 4594 4595 4596 4597 4598 4599 4600 4601 4602 4603 4604 4605 4606 4607 4608 4609 4610 4611 4612 | } } } } sqlite3ExprListDelete(db, pEList); p->pEList = pNew; } if( p->pEList ){ if( p->pEList->nExpr>db->aLimit[SQLITE_LIMIT_COLUMN] ){ sqlite3ErrorMsg(pParse, "too many columns in result set"); return WRC_Abort; } if( (elistFlags & (EP_HasFunc|EP_Subquery))!=0 ){ p->selFlags |= SF_ComplexResult; } } return WRC_Continue; } /* ** No-op routine for the parse-tree walker. ** |
︙ | ︙ | |||
5221 5222 5223 5224 5225 5226 5227 | ** is not a join. But if the outer query is not a join, then the subquery ** will be implemented as a co-routine and there is no advantage to ** flattening in that case. */ if( (pSub->selFlags & SF_Aggregate)!=0 ) continue; assert( pSub->pGroupBy==0 ); | > > | > > > > > > > | 5222 5223 5224 5225 5226 5227 5228 5229 5230 5231 5232 5233 5234 5235 5236 5237 5238 5239 5240 5241 5242 5243 5244 5245 5246 5247 5248 5249 5250 5251 5252 5253 5254 5255 5256 | ** is not a join. But if the outer query is not a join, then the subquery ** will be implemented as a co-routine and there is no advantage to ** flattening in that case. */ if( (pSub->selFlags & SF_Aggregate)!=0 ) continue; assert( pSub->pGroupBy==0 ); /* If the outer query contains a "complex" result set (that is, ** if the result set of the outer query uses functions or subqueries) ** and if the subquery contains an ORDER BY clause and if ** it will be implemented as a co-routine, then do not flatten. This ** restriction allows SQL constructs like this: ** ** SELECT expensive_function(x) ** FROM (SELECT x FROM tab ORDER BY y LIMIT 10); ** ** The expensive_function() is only computed on the 10 rows that ** are output, rather than every row of the table. ** ** The requirement that the outer query have a complex result set ** means that flattening does occur on simpler SQL constraints without ** the expensive_function() like: ** ** SELECT x FROM (SELECT x FROM tab ORDER BY y LIMIT 10); */ if( pSub->pOrderBy!=0 && i==0 && (p->selFlags & SF_ComplexResult)!=0 && (pTabList->nSrc==1 || (pTabList->a[1].fg.jointype&(JT_LEFT|JT_CROSS))!=0) ){ continue; } if( flattenSubquery(pParse, p, i, isAgg) ){ |
︙ | ︙ |
Changes to src/sqliteInt.h.
︙ | ︙ | |||
2398 2399 2400 2401 2402 2403 2404 | }; /* ** The following are the meanings of bits in the Expr.flags field. */ #define EP_FromJoin 0x000001 /* Originates in ON/USING clause of outer join */ #define EP_Agg 0x000002 /* Contains one or more aggregate functions */ | | | 2398 2399 2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 | }; /* ** The following are the meanings of bits in the Expr.flags field. */ #define EP_FromJoin 0x000001 /* Originates in ON/USING clause of outer join */ #define EP_Agg 0x000002 /* Contains one or more aggregate functions */ #define EP_HasFunc 0x000004 /* Contains one or more functions of any kind */ /* 0x000008 // available for use */ #define EP_Distinct 0x000010 /* Aggregate function with DISTINCT keyword */ #define EP_VarSelect 0x000020 /* pSelect is correlated, not constant */ #define EP_DblQuoted 0x000040 /* token.z was originally in "..." */ #define EP_InfixFunc 0x000080 /* True for an infix function: LIKE, GLOB, etc */ #define EP_Collate 0x000100 /* Tree contains a TK_COLLATE operator */ #define EP_Generic 0x000200 /* Ignore COLLATE or affinity on this tree */ |
︙ | ︙ | |||
2422 2423 2424 2425 2426 2427 2428 | #define EP_ConstFunc 0x080000 /* A SQLITE_FUNC_CONSTANT or _SLOCHNG function */ #define EP_CanBeNull 0x100000 /* Can be null despite NOT NULL constraint */ #define EP_Subquery 0x200000 /* Tree contains a TK_SELECT operator */ #define EP_Alias 0x400000 /* Is an alias for a result set column */ #define EP_Leaf 0x800000 /* Expr.pLeft, .pRight, .u.pSelect all NULL */ /* | | > | | 2422 2423 2424 2425 2426 2427 2428 2429 2430 2431 2432 2433 2434 2435 2436 2437 2438 2439 | #define EP_ConstFunc 0x080000 /* A SQLITE_FUNC_CONSTANT or _SLOCHNG function */ #define EP_CanBeNull 0x100000 /* Can be null despite NOT NULL constraint */ #define EP_Subquery 0x200000 /* Tree contains a TK_SELECT operator */ #define EP_Alias 0x400000 /* Is an alias for a result set column */ #define EP_Leaf 0x800000 /* Expr.pLeft, .pRight, .u.pSelect all NULL */ /* ** The EP_Propagate mask is a set of properties that automatically propagate ** upwards into parent nodes. */ #define EP_Propagate (EP_Collate|EP_Subquery|EP_HasFunc) /* ** These macros can be used to test, set, or clear bits in the ** Expr.flags field. */ #define ExprHasProperty(E,P) (((E)->flags&(P))!=0) #define ExprHasAllProperty(E,P) (((E)->flags&(P))==(P)) |
︙ | ︙ | |||
2704 2705 2706 2707 2708 2709 2710 2711 2712 2713 2714 2715 2716 2717 | #define NC_PartIdx 0x0002 /* True if resolving a partial index WHERE */ #define NC_IsCheck 0x0004 /* True if resolving names in a CHECK constraint */ #define NC_InAggFunc 0x0008 /* True if analyzing arguments to an agg func */ #define NC_HasAgg 0x0010 /* One or more aggregate functions seen */ #define NC_IdxExpr 0x0020 /* True if resolving columns of CREATE INDEX */ #define NC_VarSelect 0x0040 /* A correlated subquery has been seen */ #define NC_MinMaxAgg 0x1000 /* min/max aggregates seen. See note above */ /* ** An instance of the following structure contains all information ** needed to generate code for a single SELECT statement. ** ** nLimit is set to -1 if there is no LIMIT clause. nOffset is set to 0. ** If there is a LIMIT clause, the parser sets nLimit to the value of the | > | 2705 2706 2707 2708 2709 2710 2711 2712 2713 2714 2715 2716 2717 2718 2719 | #define NC_PartIdx 0x0002 /* True if resolving a partial index WHERE */ #define NC_IsCheck 0x0004 /* True if resolving names in a CHECK constraint */ #define NC_InAggFunc 0x0008 /* True if analyzing arguments to an agg func */ #define NC_HasAgg 0x0010 /* One or more aggregate functions seen */ #define NC_IdxExpr 0x0020 /* True if resolving columns of CREATE INDEX */ #define NC_VarSelect 0x0040 /* A correlated subquery has been seen */ #define NC_MinMaxAgg 0x1000 /* min/max aggregates seen. See note above */ #define NC_Complex 0x2000 /* True if a function or subquery seen */ /* ** An instance of the following structure contains all information ** needed to generate code for a single SELECT statement. ** ** nLimit is set to -1 if there is no LIMIT clause. nOffset is set to 0. ** If there is a LIMIT clause, the parser sets nLimit to the value of the |
︙ | ︙ | |||
2774 2775 2776 2777 2778 2779 2780 2781 2782 2783 2784 2785 2786 2787 | #define SF_NestedFrom 0x00800 /* Part of a parenthesized FROM clause */ #define SF_MinMaxAgg 0x01000 /* Aggregate containing min() or max() */ #define SF_Recursive 0x02000 /* The recursive part of a recursive CTE */ #define SF_FixedLimit 0x04000 /* nSelectRow set by a constant LIMIT */ #define SF_MaybeConvert 0x08000 /* Need convertCompoundSelectToSubquery() */ #define SF_Converted 0x10000 /* By convertCompoundSelectToSubquery() */ #define SF_IncludeHidden 0x20000 /* Include hidden columns in output */ /* ** The results of a SELECT can be distributed in several ways, as defined ** by one of the following macros. The "SRT" prefix means "SELECT Result ** Type". ** | > | 2776 2777 2778 2779 2780 2781 2782 2783 2784 2785 2786 2787 2788 2789 2790 | #define SF_NestedFrom 0x00800 /* Part of a parenthesized FROM clause */ #define SF_MinMaxAgg 0x01000 /* Aggregate containing min() or max() */ #define SF_Recursive 0x02000 /* The recursive part of a recursive CTE */ #define SF_FixedLimit 0x04000 /* nSelectRow set by a constant LIMIT */ #define SF_MaybeConvert 0x08000 /* Need convertCompoundSelectToSubquery() */ #define SF_Converted 0x10000 /* By convertCompoundSelectToSubquery() */ #define SF_IncludeHidden 0x20000 /* Include hidden columns in output */ #define SF_ComplexResult 0x40000 /* Result set contains subquery or function */ /* ** The results of a SELECT can be distributed in several ways, as defined ** by one of the following macros. The "SRT" prefix means "SELECT Result ** Type". ** |
︙ | ︙ |