Webinar on the evaluation of policy research

Next Wednesday, September 21st at 1:oo pm GMT I will be presenting the approaches and methodologies on the evaluation of policy research documented in “Who Used Our Findings?” at a dedicated webinar through On Think Tanks:

Can policy research be evaluated and how? The case of American foundations and think tanks

The webinar is free and there are still plenty of tickets for participation in the discussion that will follow the presentation.

Incidentally, the On Think Tank School is offering a long course on M&E for think tanks starting on September 26!

Webinar outline

The webinar will focus on the approaches and methods used by US foundations and think tanks to evaluate policy research –and its impact.

To do so, it will introduce the general practices of think tanks, but also the most innovative practices identified in the expert interviews.  On the one hand, think tanks tend to promote their presence in terms of how many products they put out, how many times they are quoted in the media, or how often do they appear before congress. Slowly but surely, foundations are attracting them towards more ambitious attempts to understand influence that usually include an increased interest in the research uptake of core constituencies, but also the gravitation towards more explicit advocacy efforts.

Outcome Evaluation of Policy Research: Recommendations and Best Practices for Funders, Researchers, and Evaluators

The recommendations and best practices that follow have been excerpted from the last chapter of my Master’s Thesis ‘Who Used Our Findings: Framing Collaborations between Foundations and Think Tanks through Practices in the Evaluation of Policy Influence’, defended at Georgetown University on April 22^nd, 2016. They were extracted from a total of 22 semi-structured interviews to experts in foundations, think tanks, and independent evaluation firms.

The study set out to examine the collaborations between foundations and think tanks in order to identify if the former are driving the evaluation of the latter, and to determine whether and how the influence of public policy research is being assessed in that intersection. The motivation for the study was grounded in a paradox: while many collaborations between foundations and think tanks are longstanding and well known, there has been to date no attempt at framing how they are affected by the increasing reliance of organized philanthropy on impact and outcome assessment.

In contrast with that general landscape, the main hypothesis of the study that think tanks supported by foundations are more likely to conduct outcome evaluation, has been supported with the identification of four case studies in which foundations can clearly be attributed as the driving force for the construction of monitoring and evaluation capacities and the conduction of evaluative research. When think tanks build such systems, they are significantly improving their ability to make a persuasive argument about their influence and contribution to the policymaking environment; and this argument is an important part of their fundraising strategy. At the same time, the Think Tank Initiative and other examples of think tank evaluation may have opened the field to studies on the impact of prominent think tanks, like the Brookings Institution, the Bipartisan Policy Center, and the Pew Research Center, that have recently been commissioned by the Hewlett Foundation.

Recommendations

1: Learn from the field of strategic communications

One of the most recurring pieces in attributing value to the contributions of think tanks was the consideration that evidence ought to be packaged strategically to stay influential. Some of the interviewees, like Justin Milner at Urban Institute, mentioned the production of visualizations, videos and blog posts.

Jackie Kaye highlighted the benefits of connecting policy research with the lessons of strategic messaging and communications from a more experimental perspective:

If you have policy research findings, wouldn’t it be great to do randomized control mini-experiments, where half of the people you want to influence get a two-page summary of the findings, and half of the people get a video and a two-page summary of the findings? I think that the work to evaluate policy research needs to get a little more creative.

(Personal interview with Jackie Kaye, Wellspring Advisors, February 22, 2016)

Similarly, Julia Coffman saw that the “wave of the future” may lie in the power of digital analytics for capturing “audience exposure to information and electronic consumption”. Even without advancing that far, some of the tools of market research that gauge behaviors, attitudes and beliefs of target audiences could be adopted more often by public policy research organizations.

2: Think about audiences

As a design principle, the focus on users can be adopted in the earliest phases of research design by addressing the information needs of specific audiences and concentrating dissemination on them.

A recurring recommendation linked to the above is to focus on the users of information. This recommendation dilates into multiple dimensions, as the information consumed can consist in evaluation findings, or in the policy research project; while intended users can be internal to the organization, members of a partnering organization, or decision makers. For that reason, the recommendation to understand the work culture and information needs of users can take many forms. Some organizations focus on understanding the needs of policymakers at the national and local level; others have focused on fostering the understanding of how policies at the federal level constrain the possible decisions of other policy makers; while still others have in mind what can be done to make evaluation findings more useful beyond the context that generated them.

As a design principle, the focus on users can be adopted in the earliest phases of research design by addressing the information needs of specific audiences and concentrating dissemination on them. Additionally, that intensive focus decreases the barriers for measuring uptake in terms of opinions, behaviors, attitudes and use.

3: Map the inclusion of public policy research in larger initiatives

Some of the respondents expressed the hope that the findings of public policy research will be integrated in a purposeful and intentional manner in larger initiatives including different tactics, like grassroots mobilizations, awareness campaigns, etcetera. In this context, evaluation is seen as a test that can help identify what is the most effective placement of new information in a planning sequence.

Best practices for funders

1: Build anchor institutions by aggregation

Offer financial incentives for collaborations between grantees working in the same area with the objective of working through their disagreements and finding a common message that can be disseminated through a collaborative platform.

2: include unnatural allies in the coalition

Including communities and voices that are usually on the other side of an issue, like a hunter and an environmentalist, can decrease the ideological perception of a message and strengthen its credibility.

Best practices for researchers

1: Package information differently

Push beyond the traditional publication of information in books and pdf reports towards formats that summarize information for easy consumption: videos, infographics, visualizations, stories or shorter texts.

2: Target specific audiences

Although federal legislation is usually targeted due to its high profile, less known policymakers at the state and local level, as well as mid-rank employees in government agencies have an important weight in policy implementation. Targeting them at the right time with information that is relevant and connected to their sphere of influence is hard, but can make a difference.

3: Connect the dots

Policymakers are rarely aware of the consequences of their actions for the work of other agencies. Relevant information about unintended consequences in other policymaking communities can promote more nuanced, self-aware and responsible decisions.

4: Create multidisciplinary teams

Varied skillsets can effectively triangulate the lifecycle of a project between the phases of planning, research, communication, and evaluation.

5: Collaborate with advocates

When there are clear legislative goals, collaborate with advocates that can make the message heard in an active way. The inclusion of advocates can also facilitate the assessment of which groups and communities have been informed by a project or are using evidence to support their argument.

Best practices for evaluators

1: Avoid superimposing strict models

Although logic models and theories of change can be useful tools for identifying and testing assumptions and assigning intended outcomes to a project, many funders avoid superimposing their own model, and focus instead on offering support to organizations with similar goals, including support for capacity building in evaluation and planning.

2: Establish intermediate outcomes

Interviewees indicated how intermediate outcomes signaling changes in attitudes, behaviors, discourse and opinions can be measured with tools like public polling or discourse analysis.

Outcome evaluation of policy research cannot be tracked with legislative change only. Other intermediate outcomes that signal changes in attitudes, behaviors, discourse and opinions can be measured. For example, public polling, discourse analysis of public political debates and bill proposals that were not passed, etc.

3: Use evaluation to better understand the world

In the realm of public policy research, the assessment of influence is usually not a summative look on whether a program delivered what it promised. Much rather, it is an examination of how the surrounding environment interacts with the objectives of a program to understand whether the message has to be repeated, translated or shifted.

4: Identify and address key decision points

Evaluation is more likely to be informative and useful when it has been designed to address program-related information needs and is delivered on time for a decision. This often implies designing the evaluation and the program as interdependent processes.

5: Make Social Network Analysis part of the influence assessment toolkit:

The Robert Wood Johnson Foundation has adopted the use of SNA as a way to measure the social outreach of academics, but also to provide insights to advocacy organizations into which groups and communities are underexposed to a certain message. The references to the utilization of SNA for assessing policy influence are augmenting: see Borgenschneider and Corbett (2010, 300) and the guide published by Network Impact and the Center for Evaluation Innovation (2014).

“Right Moves” in the marketplace of ideas

Right Moves, Jason Stahl’s new book, explains how Americans debate policy through the history of conservative think tanks. This post contains a brief review of the book and an interview with the Author.

The Review

Right Moves examines the history of conservative think tanks and their impact on the market of ideas in America. When I first heard about it, I wondered what it would add to the existing scholarship on think tanks. Many experts have effectively recognized how the appearance of new players like the Heritage Foundation and the Cato Institute transformed the field of public policy expertise. If until then Brookings had been the model of what a think tank should be in its research operations and modes of influence, with the arrival of Heritage nonpartisanship (or the appearance of nonpartisanship) was relegated from a mandatory to an optional approach, and a weaker one for that matter: advocacy tanks with major investments in rapid production and marketing of ideas first obtained the favor of the Reagan administration and then took the lead in the nineties, transforming once and for all the idea of what a think tank should do to seek relevance. Goodbye books, welcome briefs.

In some way or another, this fundamental change has been recognized and analyzed by all major experts, be them historians like James Allen Smith, think tank scholars like James McGann, practitioners like Kent Weaver, political scientists like Andy Rich or sociologists like Thomas Medvetz. In fact, the thesis that the advent of this new model of conservative organization displaced the academic production of liberal thinkers, especially of those working in universities, can be found in Smith’s 1993 book The Politics of Expertise and became central in Medvetz’s 2012 Think Tanks in America. So why this book?

Stahl explains how the apparently innocuous notion of the market of ideas became an deological weapon: conservative ideologues’s ideas became respectable because they were conservative, not because they were sound and rigorous.

As I soon found out, Stahl’s book is important because it combines historical accuracy with a cunning look at the development of a concept that may seem natural nowadays but has transformed American policy debates from their root: it is the notion of a market of ideas, which justifies the acceptance of conservative proposals for the sake of ideological balance against dominating liberalism in the progressive era. The author makes a great job at explaining how that apparently innocuous notion became the best ideological weapon, how leading ideologues propagated ideas that were worth of consideration because of their ideological stamp, and not because of their soundness or rigor. What’s more: the book shows how think tanks’ contribution to that notion had a tremendous influence in Washington DC over time. In other words, under Stahl’s thesis the rise of Heritage and Co. not only displaced the notion of what a good think tank should do ––it shifted the perception of what is a valid ideological position, and what type of arguments can enter into a serious policy debate.

The rise of conservative think tanks shifted the perception of what type of arguments can enter into a serious policy debate

Stahl’s approach is very close to the ground: as a historian, he has dived into archives and reconstructed fundamental moments in the genealogy of the current think tank scene. He follows a number of policy debates very closely -on supply side economics, on welfare reform, on the momentous aggressive foreign policy that was already being elaborated before 9-11. Most of those moments describe what are usually considered conservative shops: the consolidation of the American Enterprise Institute, the ascent of the Heritage Institution in the eighties and its symbolic triumph as Alma Mater of the scores of conservatives that were sent to administer Iraq after the 2003 invasion. But what makes Right moves even more important for contemporary readers is its explanatory power regarding the fate of liberal organizations and of the Democratic Party. Instead of serving as a liberal counterbalance to the new forces, the Brookings Institution moved to the right in search of a new center that could be perceived as devoid of bias; and the Democratic Party as a whole moved to the right under the ideological influence of the Democratic Leadership Council and the Progressive Policy Institute, referred to with the unified acronym DLC/PPI throughout the book, which sought to establish a new liberal orthodoxy for Bill Clinton’s presidency under the term New Democrat. That move to the right marginalized the leadership of Jesse Jackson, more solidly aligned with traditional liberal values that connected with the civil rights movement.

The interview

Question: What’s the origin of this book?

Jason Stahl: This book was the topic of my dissertation, although it’s different in the final product. It is part of my work during the last six years, since 2009 to last year.

Q: What is your specific contribution to the existing literature? Some scholars, for example Rich or Medvetz, have already devoted attention to the importance of conservative think tanks.

JS: With this book I am really making a contribution to two big bodies of research. The one that you mention, with Medvetz and Rich as more recent representatives, is certainly one body. The other body has studied conservatism in America. And the specific approach of my contribution can be seen in my training as a historian, which is reflected in how I combine the two themes. I am talking with the tools of a historian, going to the archives and reconstructing institutional development through those sources. Other scholars have used very different methods. They are sociologists or political scientists concerned with how think tanks operate in the present. They conduct interviews and do ethnographic work. In any case, looking at conservative think tanks as sole point of inquiry is new in both literatures.

Looking at conservative think tanks as sole point of inquiry is new in both the literatures on think tanks and on conservatism

What are the think tanks that you analyze in the book?

There are two groups: the most prominent conservative think tanks that most would name, and I mean AEI, Hoover, Heritage and Cato. They are the most prominent, and they all have a place in the book, especially AEI and Heritage.

But another important part is the interrelationships with think tanks that existed before, and of course Brookings is the most important here. These conservative centers established their brands in counterposition with Brookings or borrowing from them in what is a very dynamic process. Especially because, if you look at the development of Brookings over time, it starts out as progressive technocratic, but then in the 1930s it becomes an anti-New Deal endeavor, and it was perceived as being strongly associated with big business. That doesn’t switch until the early 1950s under the umbrella of postwar technocratic liberalism. What I argue in the book is that they reoriented themselves again in the 70s, in large part because of how conservatives reacted to them and to what was perceived as a liberal bias of the postwar consensus, which had had the effect of marginalizing conservative views. So Brookings’ response was to deny their liberal bias with very eloquent gestures that portrayed them as balanced: they brought in a new Republican president and conservative scholars. This dynamic continues to this day. By design, they have been very good at constantly reinventing themselves, shifting, reading the moment of where they needed to be to have the most influence.

Q: What is the role of foundations in this period?

JS: In the 1970s, foundations became a solution for conservative think tanks. The idea that they were fundamentally biased impeded their influence, so initially, between 1970 and 1975, it was a very big solution to say: no, we’re not funded by GE or other corporate funders, but by foundation XYZ, which is more likely to be perceived as a neutral institution that funds research. An example of such funders is John Merrill Olin: an oil business man, Olin decided to start his own foundation in order to funnel money to political projects.

On the other hand, AEI’s Bill Baroody was able to obtain support from the Ford Foundation in spite of the fact that it was viewed as biased towards by suggesting that they needed a more balanced portfolio. For the Ford Foundation, supporting AEI could lend them an aura of neutrality!

Q: What is the key takeaway of this book?

I really want this book to speak to different audiences, both to academics and to a larger public. And I think it’s also very important to have a recognized narrative for this important part of our recent history. These institutions have had an enormous effect on the way Americans debate policy today, and the book can help understand how that frame came to life. Basically, by framing policy debates as a market place, conservatives were able to validate the need of particular types of conservative voices. On the longer run, this strategy has relativized policy debate, has lent enormous new power for conservatism, and has altered how Americans understand the way political arguments happen.

Conservative think tanks have had an enormous effect on the way Americans debate policy today, and the book can help understand how that frame came to life.

Q: Does your book contain any recommendations for the future?

JS: Well, this way of debating policy is just with us. At this point, there might be a need for a broader market of policies. Until now we haven’t had a broad market place, but just a balanced market place. We have two voices, instead of the many possible voices. What we can do is allow for an actually broader marketplace. It’s true that those voices may exist but lack resonance on the mainstream platforms, and on what is heard in Washington; but the media fragmentation that we’re seeing may be signaling to a change in that direction.

Q: We are experiencing a fascinating and transformative election year. How do you think the conservative movement may change in the future?

With AEI positioning itself as an anti-Trump institution and Heritage starting to assume a pro-stance, we can see an interesting institutional split this year.

JS: I think you’re right, one hears so often that we’re in front of a timely transformation that then doesn’t happen, but I think this time we’re seeing a fascinating shift in the primaries towards candidates that don’t fall within the voices that I was mentioning. And it’s interesting to see how different think tanks are positioning themselves in the primary process. What will Heritage do? It seems to me like they are starting to assume a pro-Trump position. So we can see an institutional split, because AEI is working hard to position itself as a Republican anti-Trump institution.

Follow the Jason Stahl on twitter: @stahljason

 

Master’s Thesis: How do Foundations evaluate Think Tanks?

The interest in telling apart what works from what doesn’t is widespread in the philanthropic community, particularly in organizations that devote their efforts to policy change in the United States and around the globe. With the goal of maximizing the efficiency of their grants, prominent foundations, like the Bill and Melinda Gates Foundation or the Annie E. Casey Foundation, have confronted the need for specific methods in monitoring and evaluation that can be applied to the work of the nonprofits they support.

Credit: W. K. Kellogg Foundation. Logic Development Guide (2004)

However, little attention has been paid so far to the relationships between research-oriented organizations and the foundations that support them –with good reasons: it is extremely hard to prove the impact and utilization of a policy report.

Puntualizaciones al reportaje de El Confidencial sobre la financiación de think tanks por empresas del IBEX 35

This post is entirely written in Spanish. It contains some clarifications with regards to an article published in the Spanish online newspaper El Confidencial . The article features original research published in this blog on July 2015 about the links of corporations to think tanks in Spain.

Durante las últimas semanas he estado en contacto con Daniele Grasso, periodista de datos de El Confidencial, para preparar un reportaje que incluyera mis resultados sobre las relaciones de mecenazgo entre las empresas del IBEX 35 y los think tanks españoles. La idea era incluir una visualización interactiva que mejorara lo que yo había podido publicar por mis propios medios en este blog hace unas semanas. El reportaje ha salido hoy con el siguiente titular: ¿Cómo influir sobre el debate público? 22 empresas del Ibex financian 11 ‘think tanks’

Es un orgullo que tu trabajo aparezca como pieza central de un diario tan leído y agradezco a Daniele que se interesara por el proyecto. En este post voy a aprovechar para hacer algunas aclaraciones que, por cuestiones de espacio y tiempo, no se coligen del reportaje.

• La visualización está basada en esta hoja de cálculo que he generado visitando las webs de todas las organizaciones pertinentes, añadiendo indicadores que permitían hacer un análisis de clusters, etc.

• Por lo que respecta a los datos, lo más importante es que no son 11 think tanks, sino 12. Y es que falta Ecodes, el think tank que obtuvo la calificación máxima de 5 estrellas en el rating de transparencia que Francesc Ponsa y yo publicamos en el blog On Think Tanks en enero de este año. Ecodes es una fundación especializada en problemas de desarrollo y agua. Su buen trabajo les hace merecedores de importantes apoyos financieros de La Caixa, Ferrovial o Abertis, entre otros. Lo que es más importante: su caso es un recordatorio de que la presencia de financiación de grandes empresas no es condición suficiente para la falta de transparencia ni la presencia de intereses espurios.

• Según los datos de mi investigación, las empresas del IBEX están presentes en el 80% de los think tanks españoles. 12 de los 15 think tanks analizados tenían mecenas del IBEX 35. Hay muchos más think tanks, pero ninguno de los restantes publica sus fuentes de financiación.

• Con respecto a la metodología, hay que tener en cuenta un componente interpretativo no banal. La presencia en las webs de los think tanks de logos de empresas, fundaciones y universidades se ha interpretado como prueba de que esas organizaciones aportan fondos al think tank. En el caso de las universidades y las fundaciones, es fácil que se trate, más bien, de colaboraciones para proyectos específicos. En el caso de las grandes empresas la relación de patrocinio es la más probable. Pero puede haber errores.

La entrevista

Publico íntegramente las respuestas a las preguntas que me envió Daniele, demasiado largas para incluirlas en el reportaje.

– ¿Cuánto peso dirías que tienen, a día de hoy, los TT en España? ¿Consiguen tener impacto sobre las políticas? ¿Tienes algún caso práctico de ello?

En general, los think tanks en España no están muy desarrollados, aunque se trata de un campo en crecimiento. Hay algunas instituciones muy bien valoradas y relacionadas, como FRIDE y el Real Instituto Elcano, que trabajan en el ámbito de las Relaciones Internacionales y la política exterior. Sin embargo, son excepciones al panorama general.

La pregunta sobre el impacto es muy complicada. Es lo más difícil de medir y prácticamente imposible de probar, tanto en España como en Estados Unidos. Para que la propuesta de un think tank sea adoptada a corto plazo por un equipo de gobierno y aprobada por el poder legislativo se tienen que dar muchísimas circunstancias, es un milagro más que el funcionamiento general. Hay cierto consenso sobre la necesidad de considerar el impacto de los think tanks en términos de contribuciones, más que de aportaciones específicas. Además, a la hora de preguntarse si un think tank tiene influencia una de las cosas que hay que mirar independientemente de los resultados políticos, que son impredecibles, es si ha conseguido abrir un debate que antes no existía. Hay unos cuantos casos en España. Por ejemplo, el Círculo de Empresarios ha contribuido a poner en el mapa cuestiones como la privatización de monopolios estatales, que eran casi impensables antes de que ellos empezaran a estudiarlas. Y FEDEA, por poner otro ejemplo, lleva tiempo abogando por la implantación de un contrato laboral único. No están solos, pero son una de las voces más fuertes junto a Politikon, que yo empiezo a considerar un claro modelo de think tank digital independiente.

Por el momento, casi nadie les hace caso, pero es que el ciclo temporal estándar que se considera para que una nueva propuesta sea adoptada es de unos quince años. Es un plazo muy largo.

– Resulta claro que las grandes corporaciones quieren influir a través de los Think Tanks. ¿Cómo se manifiesta este tipo de influencia?

Las empresas españolas tienen una presencia estable y orgánica en varias fundaciones que funcionan como su brazo investigador. Algunas de ellas funcionan como asociaciones empresariales, realizan propuestas y estudios destinados a servir a sus asociados, representando sus intereses ante el aparato político. No hay nada malo en ello. Cuando vemos que se lanzan nuevas propuestas para el crecimiento de las pymes o el apoyo a la internacionalización, nos encontramos ante un trabajo de carácter altamente técnico que debe ser tomado en serio. Muchas grandes corporaciones forman parte de estas asociaciones, como es el caso de COTEC.

El caso de FEDEA es parecido, pero con matices. Aunque naciera bajo el auspicio del Banco de España hace ya 30 años, tenga un perfil más generalista y reclute a académicos de prestigio para sus investigaciones. Fedea no es “el think tank del IBEX35”, como se lee a veces en los periódicos; pero es cierto que está financiado por algunas de las empresas del IBEX y que aspira a convertirse en el think tank económico de referencia en España. De hecho, su página web publicita que se ha convertido en el interlocutor privilegiado con el FMI y la OCDE. Parece que su objetivo sea convertirse en los representantes de la ortodoxia técnica.

Más en general, aunque el mapa parece constatar que existe un interés por parte de muchas grandes corporaciones en aprovechar la persuasión blanda de los think tanks para ejercer su influencia sobre el mundo de la política hay que reparar en que muchas de estas empresas, como los antiguos monopolios estatales, han tenido siempre comunicación directa con la alta política y no necesitan a los think tanks. Su presencia en el mapa se puede deber a su peso incomparable en el mecenazgo privado, especialmente en el caso de los bancos. Haría falta investigar más para poder determinar la motivación detrás de los patrocinios que hemos observado.

– Sobre el papel, los Think Tanks debería ser fuentes de conocimiento y ‘expertise’ independientes (a menos que se trate de TT vinculados a partidos, claro). ¿Crees que, en España, la presencia de grandes empresas quita credibilidad al trabajo de los Think Tanks? ¿O quizás pese más la “mala imagen” que dan la relación entre determinados TT y partidos políticos?

El hecho de que muchas de las más grandes empresas españolas estén involucradas en la financiación de think tanks no solo es inevitable, sino que no debería ser negativo de por sí. De hecho, en el mapa se ve que las empresas del IBEX35 financian a las instituciones que les son más afines, pero también a think tanks más pequeños e independientes en una labor de mecenazgo que tiene el potencial de soportar la diversidad en el mercado de las ideas, aunque ahora mismo esté poco desarrollada en nuestro país.

Lo que sí resulta preocupante es que los dos modelos de think tanks predominantes en España, es decir, las fundaciones de partidos políticos y los think tanks empresariales como el Círculo de Empresarios, FEDEA o el Cercle d’Economia en Cataluña, estén afiliados a intereses específicos que suelen tender a la conservación y al elitismo, y no cuenten con un contrapeso suficiente en el espectro progresista. En este sentido la clave, más que en la credibilidad, está en la competencia. Yo no creo en la imparcialidad de los think tanks, algo que todavía se defiende en Estados Unidos, porque todo think tank tiene una ideología, unos valores y unos intereses, motivados tanto por su misión como por compromisos adquiridos con los patrocinadores, aunque luego puede funcionar autónomamente en la gestión diaria y en el trabajo de investigación. El problema es que las instituciones más conservadoras tienden a ser más populares entre quien tiene dinero disponible para financiar. Por eso es importante promover una mayor transparencia que permita conocer cuánto dinero se está moviendo en este ámbito, además de apoyar a instituciones más pequeñas para que puedan aumentar su impacto.

Actualización: Algunas de las aclaraciones se han eliminado tras haberse introducido modificaciones en el texto del reportaje.

How influential are corporations in Spanish think tanks?

Update: The results presented in this piece have been featured in the online newspaper El Confidencial, including an interactive visualization with d3. You can read the article (in Spanish) here, and some clarifications to the report here

Access the dataset

This research question quickly stumbles against two main barriers. One is the opacity of the field, with only 21 out of 48 institutions disclosing some kind of information regarding their revenue sources; the second is the cumbersome task of operationalizing corporate influence in an effective way.

The first barrier can be overcome with some stubbornness determination. Let’s measure what we can, and conform the project to the subset of ‘somewhat open’ organizations, in the hope that results will be representative or may be improved in the future. The second barrier is much more substantial, and I fear that my response is very partial at best. My operational definition of ‘corporate influence’ is the existence of funding provided by one of the companies in the IBEX35 (the index of the 35 most important companies in Madrid’s stock exchange).

1. Mapping methodology

1. Check the websites of the think tanks with at least one star in our DIY Transparify like rating of Spanish think tanks. I also excluded party foundations, as I know that they never identify their sources of private funding;
2. List all supporting entities. This generates some noise: it’s common practice to recognize sponsors with the inclusion of their logos in the projects they support; but quite often there is no way of distinguishing true sponsors and donors from partners, collaborators and clients;
3. Cross-reference supporting entities with IBEX-35 companies;
4. Generate an affiliation matrix;
5. Upload the affiliation matrix to NodeXL;
6. Code vertices by
   • type of institution (donor(circle)/grantee (pink square));
   • node degree (number of ties);
   • industry. I used banking (purple), energy (green) and infrastructures (orange), as they are the most important with the results at hand.

2. Structural similarity

A second step I took was to classify the grantees by similarity of their revenue structure. I chose to do a cluster analysis on SPSS with the following metrics:

• Total IBEX35 funders
• Ratio of IBEX35 to non-IBEX35 funders
• Total public funders

To be honest, I played with the different possible measures until I saw clusters forming, so this is more of a heuristical approach than a solid taxonomy. However, even given the small amount of information on which this analysis is based, it’s interesting to see, at the bottom, a cluster of corporate think tanks clearly bound together and distinguished from outliers like Fundación Sociedad y Educación or Fundación Ortega y Gasset, which has as many ties to IBEX35 as COTEC or FEDEA, but a lot more ties to public institutions both in Spain and in the Americas; at the top, a cluster of independent organizations with no relation to IBEX35 companies, and a middle group of more diversified entities which includes ECODES (which was rated 5 stars); and Fundación Alternativas, related but formally independent from the Spanish socialist party.

The research question can be answered with a grain of salt. The network mapped is quite dense in spite of the fact that a lot of institutions are missing. However, since many think tanks don’t disclose their total budget or the amounts provided by each donor, an accurate measurement of the weight of corporations as a ratio of overall support is impeded. In any case, this exercise has identified a number of central actors in the interaction between corporate interest and policy recommendations and suggests the interest of many of the biggest Spanish companies in using corporate think tanks as their proxies.

Table 1. IBEX35 companies and number of funding ties to transparent think tanks

IBEX35 COMPANIES	TIES
ABENGOA	1
ABERTIS	6
ACERINOX	0
ACS	3
AENA	0
AMADEUS	0
ARCELOR MITTAL	0
BANCO POPULAR	2
BANCO SABADELL	2
BANCO SANTANDER	7
BANKIA	2
BANKINTER	0
BBVA	5
CAIXABANK	7
DIA	0
ENAGÁS	1
ENDESA	4
FCC	1
FERROVIAL	3
GAMESA	0
GAS NATURAL	4
GRIFOLS	0
IAG	2
IBERDROLA	5
INDITEX	1
INDRA	4
MAPFRE	1
MEDIASET	1
OHL	5
RED ELÉCTRICA	0
REPSOL	8
SACYR	0
TÉCNICAS REUNIDAS	1
TELEFÓNICA	7

Table 2. Think tanks with at least some degree of financial transparency and number of ties to Ibex-35 companies

THINK TANKS	TIES
CATALUNYA I EUROPA	4
CATPE	1
CIRCLE D’ECONOMIA	13
CIRCULO DE EMPRESARIOS	14
CITPAX	7
CLUB DE MADRID	0
COTEC	13
ECODES	5
FEDEA	12
FRIDE	0
FUNDACION ALTERNATIVAS	5
FUNDACION CIUDADANIA Y VALORES	1
FUNDACION ORTEGA I GASSET	12
FUNDACION SOCIEDAD Y EDUCACION	2
REAL INSTITUTO ELCANO	0

New paper on the standards of transparency for think tanks

Next Saturday, July 4 I will be presenting at the II International Conference on Public Policy in Milan, Italy the results of my research on the accountability of think tanks. The paper, entitled “Towards new standards in the ethics of policy research organizations” explores the following questions:

• How can we speak of the transparency of think tanks beyond the disclosure of funding?

Financial transparency is an important tool to uncover the influence of special interest in the work of think tanks; however, it’s clearly not sufficient if it’s not followed by more insightful research on the content of policy recommendations that can identify who final beneficiaries are and what is the ideological framing of a certain organization. In other words, the mere acceptance of financial transparency as a sufficient index of openness can reinforce the perception of compliant organizations as sources of non-aligned expertise, instead of revealing the specific forms of influence brokerage and power-knowledge interactions. But can advocates push their demands further and identify new typologies of critical information for proactive disclosure?

• Should think tanks be accountable in spite of their formal independence from both the public and the private for-profit sectors?

The notion of accountability is a very strong topic of debate among public administration scholars and practitioners. Without a commonly accepted definition, it stands for the ability of public servants and elected officials to be responsive towards citizens, but it’s a heavily context-dependent notion with severe operational difficulties. Can this notion be applied in the context of (usually ) private institutions like think tanks? What adaptations and requirements should be met for this translation to happen?

• Is current investigative journalism efficient in promoting an informed debate about the role of policy research organizations in advanced democracies?

Information covered by the media is naturally skewed towards the ‘newsworthy’. Can this feature be supportive of a nuanced discussion about the role of think tanks, or does it naturally prevent the public from distinguishing between more and less accountable think tanks?

You can read the abstract below and access the complete paper here. Also, feel free to browse the rest of the panels of this first-tier conference.

The traditional understanding of American Think Tanks as neutral sources of technical expertise free of ideological orientation has been severely damaged both by the advent and dominance of advocacy focused institutions since the nineteen eighties (Rich, 2005) and by radical critiques according to which the field of Think Tanks has relegated social and political scientists working in university departments to the margins of public debate, a move that has especially damaged leftist thinkers (Medvetz, 2012). Departing from the acknowledgement that in the twenty first century bias and ideology are not to be considered obscure weaknesses, but much rather strategic elements and definers of the set of values that underlie the work of any credible Think Tank, this paper will analyze the complex feedback between investigative journalism and financial transparency as a collaborative enterprise aimed at exposing the relation between research and interest. Both unintended consequences and positive outcomes of the investigation/disclosure alliance will be highlighted using, among other evidence, recent pieces appeared in the major American newspapers and interviews with scholars of different Think Tanks based in Washington, DC in view of attaining a more nuanced notion of accountability. Different institutional notions of credibility and creation of value will then be compared with the most advanced standards derived from the public sector (accountability) and from the business and marketing fields (non-profit branding), but specifically applied to the field of Think Tanks. The results will be highly valuable for both scholars in the field, transparency advocates and practitioners, especially if involved with new policy research enterprises.

Key words: Think Tanks, accountability, transparency, ideology, bias

Interview about the Spanish rating in an online newspaper

Ibercampus, an online newspaper specialized in Spanish and Latin American Higher Education, sent me some questions regarding the Transparify rating of Spanish Think Tanks that I did with Francesc Ponsa for On Think Tanks.

Here is the link to the full interview:

http://www.ibercampus.es/los-think-tanks-espanoles-los-menos-transparentes-del-mundo-29602.htm

Transparency and reputation of Spanish Think Tanks

Complete transparency rating of Spanish Think Tanks

This has been an exciting week. On Wednesday my article with Francesc Ponsa, Director of the Spanish Office of the Observatory of Think Tanks, was published on Onthinktanks.org. Today, January 22, McGann’s 2014 Global Go To Think Tank Index Report, the most renowned independent evaluation in the field of Think Tanks, came out. Our article, entitled ‘How transparent are Spanish Think Tanks’, is a complete transparency rating of all Spanish Think Tanks using Transparify’s framework, and a powerful diagnostic of the development of the field, the types of organizations that fall under the ‘Think Tank’ tag, and the current and still marginal culture of disclosure. Out of 48 institutions, 27 disclose no financial information whatsoever, 19 were given a 1 or 2 stars rating, and only 2 had a rating higher than two, out of five possible stars. Fundación Ecología y Desarrollo (ECODES), a small environmental Think Tank, and the foreign policy Fundación para las Relaciones Internacionales y el Desarrollo Exterior (FRIDE) lead the rating with 5 and 3 stars respectively.

The reputation of Spanish Think Tanks

A number of the institutions we reviewed appear in McGann´s Index, showing that, in spite of the marginality of the sector and the still low profiles these organizations have in Spanish media, they are well known internationally and, in some cases, are even important partners and members of international networks. This is the case of Real Instituto Elcano, that hosted the co-authors of two of the top three reports profiled in the index. Here is the complete list of Spanish institutions in GTTTI and the rating we assigned them:

• Barcelona Centre for International Affairs (CIDOB): 0 stars
• Real Instituto Elcano: 2 stars
• Fundación para el Análisis y los Estudios Sociales (FAES): 1 star
• Fundación de Estudios de Economía Aplicada (FEDEA): 2 stars
• Fundación para las Relaciones Internacionales y el Desarrollo Exterior (FRIDE): 3 stars
• Institución Futuro: 0 stars
• Fundación Alternativas: 1 star
• Fundación Carolina: n/a
• GRAIN: 2 stars (not included in our rating)
• Fundación Innovación Bankinter: n/a
• Fundación IDEAS: n/a

The reason why the last bullet does not link to an external website is simple: IDEAS, formerly the party tank of the Spanish Socialist party, was discontinued in 2013 following a scandal of nepotism, and its staff relocated in Fundación Pablo Iglesias, the official non-profit arm of the party. This is an illustration of the lack of rigour of the GTTTI report, which seems to have defective procedures for factual check and quality assurance. The rating is not applicable to Fundación Carolina because it is an endowment Foundation, not a Think Tank, and to Bankinter’s because this is a corporate Think Tank, which means that the transparency standards for independent institutions do not apply. In any case, it´s somewhat encouraging to see that the average transparency rating of better known institutions is higher than that of the complete sector: 1.4 rather than 0.7, including GRAIN. This  doubles the general average, and I consider it a reasonable indicator that the consolidation of the sector in the years to come may also result in more disclosure and stronger reputations. It should be pointed out the FRIDE, a reasonably new player, responded directly to Transparify’s first evaluation one year ago and has disclosed their funding in different brackets as a direct result of Hans Gutbrod’s team advocacy. A fantastic example of why this work matters.

AEI’s reputation as a Community of Scholars

The view of a scholar

I have no idea of the politics of my colleagues

This was one of Karlyn Bowman’s most eloquent expressions of the American Enterprise Institute’s adamant defense of scholar independence. In a very clear narrative that represents fellows as insulated researchers deciding their subjects autonomously and only occasionally sharing their expertise for larger projects, Ms. Bowman insisted her Department ‘met once [in the eighties] and never decided to meet again’.

When asked about the emerging agenda for more transparency in Think Tanks, Ms. Bowman was skeptical about the added value of both disclosing the identity of donors and the allocation of financial resources to specific projects, and considered transparency advocacy a ‘recurring wave’. In fact, she insisted, total disclosure can harm the impartiality of staffers, who are currently advancing their work with no strings attached.

I have no idea who funds me; and I’m lucky I’m not asked to do fundraising.

When I asked her what would AEI management decide to do if a hypothetical foundation were to donate a large sum under a set of requirements, including the celebration of events on predefined topics, Bowman was categorical: ‘we don’t do that’. With a base of about 1,000 between corporate and individual donors, plus its endowment, AEI has preferred to stay small and to do research that is relevant in the long term, as opposed to more advocacy oriented organizations who are ‘harming the environment’.

The institutional view

The notion of credibility that the American Enterprise Institute pushes to embody is unique for several reasons. Firstly, there is an interesting combination between the claim of non-partisanship and rotund ideological embracement: AEI is the conservative Think Tank. AEI´s form to balance this dichotomy has been to promote the atomistic autonomy of fellow researchers as a form of decentralized neutrality, and the stress on the negative impact of disclosure offers a slamming response to recent investigative efforts to put the work of Think Tanks under scrutiny. In fact, the Washington Post´s recent piece on Brookings can be used to measure the difference in their reputational brands. According to the report, Brookings donors often buy issues on the agenda (and this doesn´t imply that either research outcomes or impact enter the transaction). In contrast, AEI donors allegedly support the institution and refrain from targeting impact on the institutional agenda.

Although it still is hard to imagine how the public may contrast the accuracy of this framework without information about who the donors are, AEI´s Research Integrity statement does a very good job in defining an alternative to institutional credibility. I recommend reading the whole document, which is incredibly well written and opens with the lines that follow:

AEI’s operations are financed by donations from corporations, foundations and individuals and by investment earnings from an internal endowment. The Institute does not perform contract research and does not accept government grants. Its research agenda is determined by its president in consultation with its trustees, scholars and fellows, and academic advisers; the substance and conclusions of its research and publications are determined by the individuals conducting the research.

The statement also reinforces its tradition of impassion in the face of everyday politics:

AEI is generally prohibited from attempting to influence legislation in the U.S. Congress or other legislative bodies. Legal requirements aside, AEI has important reasons of its own for abstaining from any form of policy advocacy as an institution. Policy research of the kind AEI specializes in—emphasizing empirical analysis, intellectual depth and originality, unflinching criticism and concrete proposals for reform—is an inherently individual activity, best pursued by a single scholar (or a pair or small group of scholars) rather than by a committee or hierarchy.

Finally, the in-house recipe to control for conflicts of interest lies in four methods: diversification, disclosure, reputation and intrinsic quality. In particular, disclosure is defined as follows:

AEI scholars and fellows are required to disclose in their published work any affiliations they may have with organizations with a direct interest in the subject of that work. In the release of its products, AEI also discloses relationships in which donors have a specific material interest. AEI scholars, fellows and officers provide annual reports to AEI’s president listing all of their outside activities; the president then provides a summary report to the Nominating and Governance Committee of the AEI Board of Trustees.

Even if we weren’t to share AEI´s beliefs on donor confidentiality, their notion of integrity is in my opinion very articulated and worth of inspiration in other parts of the world.