From a676b2aa10a9a7d5d5ef76e52a8393c2eb6a5db0 Mon Sep 17 00:00:00 2001 From: Anton Kaliaev Date: Wed, 21 Jun 2017 14:57:11 +0400 Subject: [PATCH] add indexing proposal [ci skip] --- docs/architecture/adr-002.md | 90 ++++++++++++++++++++++++++++++++ docs/architecture/img/tags1.png | Bin 0 -> 10446 bytes 2 files changed, 90 insertions(+) create mode 100644 docs/architecture/adr-002.md create mode 100644 docs/architecture/img/tags1.png diff --git a/docs/architecture/adr-002.md b/docs/architecture/adr-002.md new file mode 100644 index 000000000..3c1f51b6e --- /dev/null +++ b/docs/architecture/adr-002.md @@ -0,0 +1,90 @@ +# ADR 2: Indexing + +## Context + +In the light client (or any other client), the user may want to **subscribe to +a subset of transactions** (rather than all of them) using `/subscribe?event=X`. For +example, I want to subscribe for all transactions associated with a particular +account. Same for fetching. The user may want to **fetch transactions based on +some filter** (rather than fetching all the blocks). For example, I want to get +all transactions for a particular account in the last two weeks (`tx's block +time >= '2017-06-05'`). + +Now you can't even subscribe to "all txs" in Tendermint. + +The goal is a simple and easy to use API for doing that. + +![Tx Send Flow Diagram](img/tags1.png) + +## Decision + +ABCI app return tags with a `DeliverTx` response inside the `data` field (_for +now, later we may create a separate field_). Tags is a list of key-value pairs, +protobuf encoded. + +Example data: + +```json +{ + "abci.account.name": "Igor", + "abci.account.address": "0xdeadbeef", + "tx.gas": 7 +} +``` + +### Subscribing for transactions events + +If the user wants to receive only a subset of transactions, ABCI-app must +return a list of tags with a `DeliverTx` response. These tags will be parsed and +matched with the current queries (subscribers). If the query matches the tags, +subscriber will get the transaction event. + +``` +/subscribe?query="tm.event = Tx AND tx.hash = AB0023433CF0334223212243BDD AND abci.account.invoice.number = 22" +``` + +A new package must be developed to replace the current `events` package. It +will allow clients to subscribe to a different types of events in the future: + +``` +/subscribe?query="abci.account.invoice.number = 22" +/subscribe?query="abci.account.invoice.owner CONTAINS Igor" +``` + +### Fetching transactions + +This is a bit tricky because a) we want to support a number of indexers, all of +which have a different API b) we don't know whenever tags will be sufficient +for the most apps (I guess we'll see). + +``` +/txs/search?query="tx.hash = AB0023433CF0334223212243BDD AND abci.account.owner CONTAINS Igor" +/txs/search?query="abci.account.owner = Igor" +``` + +For historic queries we will need a indexing storage (Postgres, SQLite, ...). + +### Issues + +- https://github.com/tendermint/basecoin/issues/91 +- https://github.com/tendermint/tendermint/issues/376 +- https://github.com/tendermint/tendermint/issues/287 +- https://github.com/tendermint/tendermint/issues/525 (related) + +## Status + +proposed + +## Consequences + +### Positive + +- same format for event notifications and search APIs +- powerful enough query + +### Negative + +- performance of the `match` function (where we have too many queries / subscribers) +- there is an issue where there are too many txs in the DB + +### Neutral diff --git a/docs/architecture/img/tags1.png b/docs/architecture/img/tags1.png new file mode 100644 index 0000000000000000000000000000000000000000..a6bc64e81568566d13c87996f2e86a41bcaa055a GIT binary patch literal 10446 zcmeHt2{7C5*Kc%F^|Q3vsw%oG{h_KFyKbr|8fyts6h$QnQn6JxZEbCBwO8#FMG4|7 zZ7r#Nj}rN+eTj%I2_oFIzx#jhd+*GhJMW!&XYM<5Gs%3P<$RuV&U4Q9JkL3kS9gqb z1@;}?2Z2BY^lsfSfk1YG5C~7z-ksc$XQ7=K?qiqBRl}R;AQ0aR5Xjs;2t@5Y1S0C0QUlZA;&$J;ZKkum zy*)BA5*ixHrIeSKuc)Z#>FHs!*$oX1+1c4vR#q`FF{`VqM~)nso143F;|3Cmtgo*p zlgV0ITB@q5Zffqu!gU#vh?>DZVk|`K%r>;HkplOzkvpHK1UuS7+n{91vas7*h zg@vA;p6AY;i;Ii9ckkYtH*ebJSmecZ$FLM-J+q)3xNC%Aa;Mi_|L};fprISqH={!Tru&`xM{0o8+60a(D1yfR2({UL zI$jsK`tvv#=-z<5+%0j7+u`%PWrc=7jwJv7@#y(JjfFtY;`DA@HG4EfOXW)@->gi7 zsr)SX4E}9Gj;H;QBrAy-&g&s7)!){BxfCDgv0#YY+BVipm4%#!8}4}cII@0@9wb?D zWyiU+g72-?pD(=Ta{15uXqcPbzV@o03Irtb}3XZ%Q9ni z?OI3j>e|=S|9|=a$mS4R;Timddl51f9UHinPnVo5mOherlU-a+*If&|a{SL`M#tl7 z&$h5B$9(G*UT>L?TFhTAncowARO-#k-!>b`oGkhfZ$e1+QjlS8`^pxWNVDs}q`wYD zi~T!_?AHef;#7(3VYTV&2Wm)RSZUu$-1*xv#G4X-+n&@qzW&nsu$1s0Um0qI~B)ETJ z4G6c^);ccoM+)5*xPGch0XoLLl3^hm?mur^vj>|1e!dU)#YZOF^cuN1^@DOnQc`dJ z5#-dQ+-&kgWGp8(ky@PFCz%EPyPq@lf-%MC<0OhPpb~lS|M6m$ioa>>gPciCthTB8 z0`iApx(+O6zB5I%ucloDZ-L*Vg-%_ucV(yAGha6@3r(ivwHyAzL8uQ(A2R^%Ee5XmG`SVNOKI39o(Wze-3y>ro^Al1I z&klltR;nD7xpiFi$hpYd?vB!FF>ey znkKw_qS6Fhj>*59pINJBgB@P`>ep6$trsewW9eNEcRiF}$5TA!&=r8Y+^at;d!AO4 z=Or!+Hu}-SJEFGCZr|)wAzjmIkUWFJZVCr25TDyHjh`tojA(C4riwp)tSaQ#>Xp)D z&YqzHIFhhyT~TNWp=&E5%C8)kpWmFCosl^HENA-=P_jMi%r{Cguduh+bboosqa$#(2w(mJ=EPhtp^X zqs`h@ZTV4$HEs9-sXUDRl#L2@`eKHX%FmUxu&eaKX$|1~$EsJZDPrRU0a#+i{e*>L z&3w%DS6H`cz{RhlP%RPdI`~w6ne(W#R?9EZ?T*v-CDoN$jK~V`Vm5eMLv{_>7@M&0 zNi!dsE}!o8Xpis9_7v;n^(%2<)Sz2_^MZ>XTDDv-;`b#7<7++ntKG4jk-Q(v!L#=~ zCbw!N)*}nH2TFeQn|c7!MiEY&Gvn22D#nxvscdKVmHM;&p1>k#5x{1hNtLoQA4YTx zNmRf3JFhGx=})LnZLDpzw#EqIsC@mIA6o9blK>0bqi=q(4KzA6E@E0!|zD`@~iCisM|#&LSN=Sm0NcJ*v*uH;RVv|y80S{nypj+jeM zU*1@WW?L*&&JQ2WZy8PCU%U9VERU*=HMMOB?7w4Yn&GhEC~AS}e;ZJFz2K5IEki%g zU_Gk4C0O*nqVz1o=-ssejswlOe6fB&Y6ynZY~(HCfhsV_lp5THbVZhy_3=Ub-;g56x3BeI6b|6EUhV? z>O73VhrnRT=N3woTlN5%83;QN@uMeN;P;uCatKkMYUD9&ywui%?;1iP=SMhbgSdby zy-&&*2cfV`AKCsN+KVSgZ5de)y8b+|kRp9;P#Af4Wno&{LO!7tQkNS4KmsI_su+_i&!!5LWH)p&B{R zkjc|wKzKtC5Q|xzvaF`HPE#HoP1POV99C9ZK)p!AA%4irGQiYfm5aOzoSzWxVgFrq zDXzZ$cG=14k*S#t*PK2>l_kY??Jo&=c61U7l(+ca11FQtEX*lxl}6jq+h}d@fncoqsChjd$--NY zPHEoW{MnbZJ!@>LBkM)Dl*KWq3$dedUgHRuv=#E8)@y3{M9|F$0zaaHNzEHOgDmIl z2tf_^tG@t4xyP5wEU;}rsuL$45YmrhI|-RbmRX55O<%SxVk1bZ;Ov)FOqMRa*!lAFGp zWYGOukJKTrXSNq}{GNKctFP-y4b*1YyxU4Fz%Yub=&_vxPyhHJpuI$uCejefDFrQl z1ia(M(#3LdPp0EEN&AVzOo~WniGZoaQ?Qhi_mguBVJt!m!u%!5Y_Az+J^*X#V$zd- z+ESc{dCjtyza@^Y?CZ_4ovv(m(27f#J83fXWpZ>2R%~c3z>k#uevspFW&!25c>URQ zoThN(Z|>=0lAGj%yjVO5p6^CrEHs5~P)O<+#18Ddy|kZB`ZcXN%3TpR)^HWlhVd5U zhtsEiB|O(KAMKcxHr3O5IdSXDzusFB#MM`*6lnGd3D5{tWsaoN(@HM>AVX<7U)%DsWPjkv@Xn|>P6>tF>Hetd zXmz4nTA4m`l@wb=80jifPguWND@*m0E96)?EulHMj>lI18FIm)Y!cGsS>WmqQV#c#m17UpP1 zMhYtQ_Vj?x3RIkli`I20Z|~Il?oHU~ zBqSg4Pqt~*#ktN+FE8=I-o2iGHYr1_^gWoc>-T&a<6Oe&nLWSTQLk}YZ=&||hCJq` zt&Pq7Zo(YOTjlL1Gl;Z=_#eo^UuB3GX&CdDO;jiG;4=|9h>bFlvG56OB#rVwBs*XL zTC&R*9>xoKq@v4^AATl)orc&b=rR^kz?syYp1`qQb-FR`p@bCV*Eeo^1v>yIa)~bx zS0&{~gah9yLw;Q%t`_qFXh?C&DXdF{HuG8-~9lEeA@x=>^=NnizJ?ZrYNaMX*cAa{0C^_ zKaG4`P7#{-Ys}O!z#OWer*!r+3kkD;Pak?L-SiMxIhmpMy5Q%LA~gG^IjLgQzEURh z%jMlDv(vARGA=78Uc8(V>{=F^XSyHqYw3MH=FmWFmVc~BV@-C?Q3GEwQx8^7mZ93m z2v*+s&|g^hleXvdm(PvkY!2wu`M%(Jq}JA>u5{z8*^rj9~uW6f@j+i0eHnfL2L~&yVqH&;n71Xm*O^}O3Scyo#U2%6btSrLDO3q1C z^ug4!tbW~BgG4uJ0QqtojqmanF2+~6yw^u#%bVt+K=XFPT4K~d1{4zA_&0Fgmi)F7 z^)+EwBE29l#uv26cA$}1V|t29vrrCc{%om751f5ssJ(V4L^2vJ z@7`>hc1$7sU2dmAf#dkup_@QaldsJ{CaBo>TwU6k)nZ=hYr*r#EhPXZn|^9#Wo9@D zU+$z1oFem}l40}~ruboT&UBg^!wqjy*^@l8PTH3~rV0yA3{0MMU0Y`RxIZ^unk`#y z4SzNt(!S<&clL=W+FYl4fhO0Nhbhi8-@aLWy&6_sto#?|VJ@rvH+(W1n~(`MUhI6O zMYbL_4Xnu8EU3yk9qC;k<5R&&x+9_BI$)iv40DI}Ft>|mgkX~-@s>}N*ol^fK|U#4 zO~Ed9aEF#gp|=IbyH7m_z=#JS;nFsA1*u|;GvyMgH}8Xdr|I0syqwt1gZDp`!V0pR zB(1W3xwv^bFmE$ZNG%{Xb)+&4_g_MU*v)A2CSO#;Uuh5Bc10n<)s(#l*sL z!!X_+rr6~bkFJ1f(0@oO=9PJ6cF$!E3ozczUAe-Ra$s=$(-(=Rp<^@c5_$v4J_&hW zrjJE@o1^Ks!v=F|?)4hHu4p_gaehI%FTwyJH7DU~hI7<6f70vR6_5`GXZgO zFsq!|hcx(WTB(C(@=%D>aG6f|sR|ud9bn?WO69?vcw=1J_Ui zdHbo=%Q>^TJ)gijDJzM7yll7XNOyy8pLb?EXi<63yYOLkF6O##DHLtD#CHiA{>_Ex zlOVUg`1mnsz^Lmz)bYc*uUE)IO}9+0dSfEIdV7L-GdzCjX*h$7(Aet#T!LSHzk$pzBax}Cyr7> zSab3tAuIJHSkbOkJ3Um+FQY8?@72o2RRgpz(kD!Z()UJ{nD~{(IL<1{e@pi&J#MYH zkiE7d1SeJqb_M7F>)?wFu?}rXo~iR``Jo_nky6NB-x@(l49Z@in=^thai!T@=vS=U zwx|9iL9o~9t<+S})Hdq%?LafS<<;AtucBsx?192Q;pW;+wh)mP%%G9J74fXT02BJ~ z`tXo%Rkwf;%r&~D_CpEEzAJQ9tc%mh`Ni6OK(aSW#nOIqdY!|E->&lYbcX=ajE*sD za8w3C-@YTMQDAjyfLq}Ccc?f)9xdH%rSoNY29^<1Hepk0g+ zc4_*+@&Z=6J!XK9;Kju-j!=kT&C&}{m$4O1`8EmxI{Vn#oav{_Z1w(_+3E%nnzzRJ zxN8)Hko9gw!c!HwJwP;7PGS~Y_%ubSZWOu>U&` z-efe#c?_4B3Muvhwp&9qKhH|Vo-4jVe@UNw=tT8c=Zpvitn0&Vh!yw!%4DZ84!+D)%3EN?XvrN-Pu)vvuI;T zCouVfZiaA{ThnKxX9R$Owc+wrt^2vG72AubSHaxKzZJncp_Vp77NLG%#dPCC4$nzN z&2ku+w4!mFrnnhShurYyGAL&SAhwn_G;s~9OvdpI6x}vpD-=`*5Ck;7Hi*4BIla)& z#&yyTmlSQ&xeE;hOuPi`7a#P0h*44h&2x&m!4%MBFD4~`U3ej>mv%pQA3Zt;T^&{! zt{aX7P{F6LQ?D;TB=&C20LS$UfzZHU1~#A`Jp)K? zqkT50CEEj^*9f>MH@O2JPX|2tvW%A-Rp<=}l*ZQSP#P2`d?j z<5(>UuQCyD&d+h4v$u)41*~GUWTZ>|li3tq*$fXAOFFBgi|mLcWNc}$@HjdzkdBY} zl`iCvo{RIb=wX{~B(t}rX45}l9%13%#tj8K%ZZ>=6`Qm09)zy8rP6D%g>X;7;453p z>vF4FEtac%h)SkXcXXHf4IuMe$TO*|t+1z#E8A^-Y?=V~8oN19Aq#a1u5LXK7VBA$ zn;E2H2+QkbK;jA%+NoWYxB{a?S2z4NGP%YQrKW_eLq8HYYnC6k*=JS-P3gp6_NMM> zP@~C&674Ppg%I^F_GFl?<2){SWo#uCq$9M)adr#TAav?9KEzF#zezUc5-qZ*G6zcAYFh_GCV>~27Hhp@MuHf^pL zr*q0JIpygY(A8jwQ1z$DH;8KcU|pr+bGAILVbhKB?$yFQEp!#jd`$7`D?}FZ_R>!t z&54hF*OA(RYm9-Xg_e?5D94Xhbj`m^)8y;;ZDaQV*>kqa!~rLtdG9Ydqv2Moob&PC z8jun8C7s+0Qm>g-ObRq+{K9+l&)Q zsZGjSMU#pQlE~~edWCQSrg+Z6yd};dv*4-1rcK2JX+mcbVF~M1*iUz2bcEOGJm_(@ zQ9S)?#8I|?*lMQt(=aKej7`j!$nko2z8^2MF>h`~GTvWf`JAqb-tTn1S*BmqGoH3_ zsZMykgnU`5#jgu*{EAw##0C**T%Ue6SM=ztxp>|9`4XzEw`lH{ex!tsZ1Jl{?~C$2 z1wEy)%CMq*uw&JBwKSqAXJ7j2Vwz!fg@IQc?w9 zGpKj|NlnG`Z&zJtHp?m5mB#O0lun#O1>TjmMlEtA? zM&Q9+0TR#VXvz96el8N2lAgM-X*j+1Y|L!ZMSPi&z9r~C2d59+*D(-~x?>eb@*sSt zYu1XI^EFx-(!eL0!$*^y(H}4#`;0skB4vtAQSiNTqNO=y%~fkD;F-9%_PTbJ{t63; zxJ^3;=x2Cr<*it+W&9;SEF4$w)&@)%rBUSomoFqzIN+ z>CqUPR39+uiJPhFTH-mT?6(~R;JsuX0?9u@XoKI%^c@vkC+udo;#S^noE$Z;Bzw4q zO)wc9wr>8TNqz_)M}3=+%2-Zaf6yO}RiL@{aQjg3y_sM2^=l`KT@iRY`qlQ~DlcXz zh-bCk(Zh&74EMgkj7oa}Zd#I2HDCO8_fd;nx8vB!f~LZ=o^rE9taOn7#*?<7+>l$% ztU5uKX2fo6$x~0jK>g)PkzdT&@gtp5k_%T2eDvvk{ZCc90$3oi2#3hK34}{$Lle-t zvY1|kj)KpmT{bTkdYziO)Ro-Kn)@{5;hxtr{3tBEBgc#>yST|~}H zco9nr!lF{>!%JSh5%uas34qdu7w28591Q6c*fZ{C^J7EPaDp96iRpn?_rP91x})LK z!9O5~;l=7|D2nCFn+8v4_?1vvx57i++XP!u=NEv!$c&n+s7dj)xh?Jmf_mH%MRto; znr>>J>#NUp3wFzqU;ES=>UW4wkzmh$QiVJIVvV4_rg1Ig6{xE9XFs_Hy_8d>4^ZP*ko*~w z2lPFh#VrN_*$CU=S;`U1)!=Unmr|-e?@U)~P60id>mTC3M^2F+}HY(Su#yX}PQs6#pU~XNFIkY`%Lf%Ifc=UYaJ}b@e zM^+uC3O+-hjVP<~be+$xG;n<4-Uk;opIb>2wQXJ}9ZGL0D|z@0>eoHeG`@Y=Yt}te zcHF_`fqA@t!ScjtK#w_dY;Z_i#&V z8BlI*t>`etP6!yte^&T%d4ro4aW6|HCJ}I5;j>)s6SUp^q zC_I>n|Ds*!5aR1=vdBn0gb+UJ61`ZTl(7Ry_-(H2BOHmkg^gegeb30S=}*>Nww|`@ z|A5v*mvLGfVPfZApE8{6$x6TUFVRW~>=wE!KhOa0^opz&t2NeBNOB4RjG|J`@KcHd!Q~{d`BevYL!r`dO+}%BCFijoR9aC_T)!5$9@_d&STkVD+dY~-KDpv@qnR|R zl)0(3zFJtMSr=WR#hKXGB4l&0kq|2%eGIBxuq6`jsNz_miV7_F%BAW9&HZib zA2W{MZp vl%Q(L7tUX}pmyOx>E4PT{}aFib>G$5|9=noYA)dOTM<1SqZ`H7?mhW$KZ<@5 literal 0 HcmV?d00001