Is it valid to iterate over every permutation of a regression specification and compute an “average...












1












$begingroup$


I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner.



Let's say I run an OLS model and the results come back significant. There is discussion as to whether the positive results hold up given a different set of control variables. However, it's theoretically unclear which control variables are actually relevant, and we don't just want to throw in the kitchen sink.



Let's say it's computationally feasible to run every single model (with every possible permutation of control variables), and we collect the coefficient and standard error of the coefficient of interest in each. We can report the proportion of models where this coefficient is significant, but is there some kind of meta-test that can be done showing that the test statistic is significant in a significant number of alternative specifications? Is this ever done?






regression least-squares







share|cite|improve this question









$endgroup$

















    1












    $begingroup$


    I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner.



    Let's say I run an OLS model and the results come back significant. There is discussion as to whether the positive results hold up given a different set of control variables. However, it's theoretically unclear which control variables are actually relevant, and we don't just want to throw in the kitchen sink.



    Let's say it's computationally feasible to run every single model (with every possible permutation of control variables), and we collect the coefficient and standard error of the coefficient of interest in each. We can report the proportion of models where this coefficient is significant, but is there some kind of meta-test that can be done showing that the test statistic is significant in a significant number of alternative specifications? Is this ever done?






    regression least-squares







    share|cite|improve this question









    $endgroup$















      1












      1








      1


      1



      $begingroup$


      I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner.



      Let's say I run an OLS model and the results come back significant. There is discussion as to whether the positive results hold up given a different set of control variables. However, it's theoretically unclear which control variables are actually relevant, and we don't just want to throw in the kitchen sink.



      Let's say it's computationally feasible to run every single model (with every possible permutation of control variables), and we collect the coefficient and standard error of the coefficient of interest in each. We can report the proportion of models where this coefficient is significant, but is there some kind of meta-test that can be done showing that the test statistic is significant in a significant number of alternative specifications? Is this ever done?






      regression least-squares







      share|cite|improve this question









      $endgroup$




      I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner.



      Let's say I run an OLS model and the results come back significant. There is discussion as to whether the positive results hold up given a different set of control variables. However, it's theoretically unclear which control variables are actually relevant, and we don't just want to throw in the kitchen sink.



      Let's say it's computationally feasible to run every single model (with every possible permutation of control variables), and we collect the coefficient and standard error of the coefficient of interest in each. We can report the proportion of models where this coefficient is significant, but is there some kind of meta-test that can be done showing that the test statistic is significant in a significant number of alternative specifications? Is this ever done?






      regression least-squares




      regression least-squares






      share|cite|improve this question













      share|cite|improve this question











      share|cite|improve this question




      share|cite|improve this question










      asked 55 mins ago









      ParseltongueParseltongue

      267114




      267114






















          2 Answers
          2






          active

          oldest

          votes


















          3












          $begingroup$

          One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.



          https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf



          I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.



          https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419






          share|cite|improve this answer









          $endgroup$





















            1












            $begingroup$

            Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.



            In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.






            share|cite|improve this answer









            $endgroup$













              Your Answer





              StackExchange.ifUsing("editor", function () {
              return StackExchange.using("mathjaxEditing", function () {
              StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
              StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
              });
              });
              }, "mathjax-editing");

              StackExchange.ready(function() {
              var channelOptions = {
              tags: "".split(" "),
              id: "65"
              };
              initTagRenderer("".split(" "), "".split(" "), channelOptions);

              StackExchange.using("externalEditor", function() {
              // Have to fire editor after snippets, if snippets enabled
              if (StackExchange.settings.snippets.snippetsEnabled) {
              StackExchange.using("snippets", function() {
              createEditor();
              });
              }
              else {
              createEditor();
              }
              });

              function createEditor() {
              StackExchange.prepareEditor({
              heartbeatType: 'answer',
              autoActivateHeartbeat: false,
              convertImagesToLinks: false,
              noModals: true,
              showLowRepImageUploadWarning: true,
              reputationToPostImages: null,
              bindNavPrevention: true,
              postfix: "",
              imageUploader: {
              brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
              contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
              allowUrls: true
              },
              onDemand: true,
              discardSelector: ".discard-answer"
              ,immediatelyShowMarkdownHelp:true
              });


              }
              });














              draft saved

              draft discarded


















              StackExchange.ready(
              function () {
              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f387998%2fis-it-valid-to-iterate-over-every-permutation-of-a-regression-specification-and%23new-answer', 'question_page');
              }
              );

              Post as a guest















              Required, but never shown

























              2 Answers
              2






              active

              oldest

              votes








              2 Answers
              2






              active

              oldest

              votes









              active

              oldest

              votes






              active

              oldest

              votes









              3












              $begingroup$

              One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.



              https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf



              I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.



              https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419






              share|cite|improve this answer









              $endgroup$


















                3












                $begingroup$

                One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.



                https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf



                I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.



                https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419






                share|cite|improve this answer









                $endgroup$
















                  3












                  3








                  3





                  $begingroup$

                  One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.



                  https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf



                  I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.



                  https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419






                  share|cite|improve this answer









                  $endgroup$



                  One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.



                  https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf



                  I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.



                  https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419







                  share|cite|improve this answer












                  share|cite|improve this answer



                  share|cite|improve this answer










                  answered 35 mins ago









                  eric_kernfelderic_kernfeld

                  2,9821726




                  2,9821726

























                      1












                      $begingroup$

                      Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.



                      In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.






                      share|cite|improve this answer









                      $endgroup$


















                        1












                        $begingroup$

                        Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.



                        In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.






                        share|cite|improve this answer









                        $endgroup$
















                          1












                          1








                          1





                          $begingroup$

                          Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.



                          In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.






                          share|cite|improve this answer









                          $endgroup$



                          Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.



                          In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.







                          share|cite|improve this answer












                          share|cite|improve this answer



                          share|cite|improve this answer










                          answered 35 mins ago









                          Bryan KrauseBryan Krause

                          497210




                          497210






























                              draft saved

                              draft discarded




















































                              Thanks for contributing an answer to Cross Validated!


                              • Please be sure to answer the question. Provide details and share your research!

                              But avoid …



                              • Asking for help, clarification, or responding to other answers.

                              • Making statements based on opinion; back them up with references or personal experience.


                              Use MathJax to format equations. MathJax reference.


                              To learn more, see our tips on writing great answers.




                              draft saved


                              draft discarded














                              StackExchange.ready(
                              function () {
                              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f387998%2fis-it-valid-to-iterate-over-every-permutation-of-a-regression-specification-and%23new-answer', 'question_page');
                              }
                              );

                              Post as a guest















                              Required, but never shown





















































                              Required, but never shown














                              Required, but never shown












                              Required, but never shown







                              Required, but never shown

































                              Required, but never shown














                              Required, but never shown












                              Required, but never shown







                              Required, but never shown







                              -

                              Popular posts from this blog

                              Knooppunt Holsloot

                              Image
                              Knooppunt Holsloot Kenmerken Wegen A37 × N34 Type knooppunt klaverblad Portaal     Verkeer & Vervoer Knooppunt Holsloot is een verkeersknooppunt op de kruising van de autosnelweg A37 en de N34 in Drenthe, tussen Emmen en Coevorden. Oorspronkelijk was deze aansluiting een afritconstructie van het type Haarlemmermeeraansluiting, zoals bij Knooppunt Hooipolder. Sinds 2002 is het knooppunt omgebouwd tot een volledig klaverblad. Als noviteit werd hier voor het eerst gebruikgemaakt van dimbare verlichting in plaats van de normale vaste tijden waarop de verlichting aan en uit gaat, wat een grote kostenbesparing opleverde. [1] Dat Holsloot als volwaardig knooppunt is uitgevoerd, waarbij alle verkeersstromen en afbuigingen ononderbroken kunnen doorgaan, is uitzonderlijk. Het betreft hier namelijk een aansluiting tussen een autosnelweg en niet-snelweg. Een vergelijkbare situatie treft men nabij Hoensbroek bij het knooppunt Ten Esschen.     Aa
                              Read more

                              Altaar (religie)

                              Image
                              Deel van de serie over kerkelijk gerei waaronder paramenten en liturgisch vaatwerk gebruikt in de liturgie Liturgisch vaatwerk Vasa sacra Miskelk · Pateen Kelklepeltje Ciborie · Monstrans Pyxis · Custodiale Vasa non sacra Ampullen · Wijwatervat Olievaatje Wierookvat · Ablutievat Paramenten Amict · Albe · Baarkleed · Cingel Tuniek · Dalmatiek · Fanon Kazuifel · Manipel · Stola Gremiale · Benedictievelum Mijter Koorkledij Rochet · Superplie Koorkap · Cappa magna Kovel Kelkgerei Bursa · Kelkvelum Ciborievelum Kerklinnen Corporale · Kelkdoekje Lavabodoekje · Palla Altaardwaal Kerkinterieur Altaar · Ambo Biechtstoel · Communiebank Doksaal · Doopvont Faldistorium Godslamp · Hoogaltaar Heilig Kruisaltaar Katheder · Preekstoel Sedilia Tabernakel · Volksaltaar Liturgische boeken Altaarmissaal Benedictionale · Brevier Evangeliarium · Evangelistarium Graduale · Kyriale Lectionarium Psalter · Rituaal Sacramentarium · Volksmissaal Overige Flambou
                              Read more

                              Gregoriusmis

                              Image
                              Een afbeelding van de Gregoriusmis uit een getijdenboek gemaakt in de zuidelijke Nederlanden tussen 1500 en 1525 Gregoriusmis, Meester van het Akense Altaar, ca. 1500-1525. Collectie Museum Catharijneconvent De Gregoriusmis of de Mis van Sint Gregorius is een motief in katholieke kunst, dat het eerst verscheen in de late Middeleeuwen en dat nog steeds werd gebruikt tijdens de contrareformatie. Paus Gregorius I (ca. 540-604) is afgebeeld, de H. Mis opdragend, terwijl Christus als de Man van Smarten verschijnt op het altaar dat voor Sint Gregorius staat. Volgens de legende had St. Gregorius gebeden om een teken voor een betwijfelaar van de transsubstantiatieleer, waarop de lijdende Christus tijdens het opdragen van de Mis verscheen om hem te overtuigen en waarbij zijn bloed in de altaarkelk liep. Vaak zijn ook de passiewerktuigen afgebeeld. Zie de categorie Mass of Saint Gregory van Wikimedia Commons voor mediabestanden over dit onderwerp. This page is onl
                              Read more