Frequently Asked Questions

Updated January 2024

INFER Overview


What is INFER?

INFER (INtegrated Forecasting and Estimates of Risk) uses crowdsourced forecasting to generate early warning and signals about the complex future that policy and decisionmakers must act upon. It involves an ecosystem of forecasting platforms that aggregate the diverse perspectives of a global community inside and outside the U.S. government.

INFER is led by the nonprofit RAND with support from Cultivate Labs.


The domain of this site is INFER-pub.com. What is INFER-pub?

INFER Public (INFER-pub.com) refers to our open forecasting platform available to a global audience from inside and outside the government.


Who do I contact if I need help?

For general questions about the project, click the "Contact Support" button below or send an email to [email protected].


What am I being asked to do as a participant?

We are asking participants to estimate the probability that various events related to science and technology trends and geopolitics will occur and to offer a rationale for their forecasts. INFER is designed to generate real-time crowd forecasts, so we ask participants to log in regularly (several times per month) to make and update their forecasts. We will also release new sets of questions periodically and encourage participants to check in when alerted of new questions. We also encourage you to support the project in other ways, such as suggesting forecasting questions or topic areas to the INFER team, inviting friends and colleagues, and posting about INFER on social media.


How are the questions on INFER selected?

INFER gets requests from policymakers about what they need to understand about the future. We then agree on a question that best represents what they want to know. From there, our team goes through a process of “decomposing” that broad question into the individual questions you see on INFER. Learn more about our decomposition process.


Can I invite my friends or colleagues to become forecasters?

Yes, of course! We encourage anyone with an interest in forecasting, science and technology, or geopolitics to apply to join the INFER project community. No specific background in any of the topic areas is necessary.

Back to Top

Making and Updating Forecasts


How does forecasting work?

To make a forecast on INFER, simply enter the probability (i.e., 0-100%) you think an event will occur, along with a rationale for your forecast. Here is a helpful article with tips on how to approach your first forecast.


How do you generate the crowd forecast?

The crowd forecast is an aggregation of all individual forecasts at any given time. Initially, the crowd forecast will simply be the mean of all individual forecasts. Over time we will transition to more sophisticated aggregation methods, such as putting more weight on the forecasts of individuals with the best track record.


Who determines what the correct answer is?

The background section for each question provides information about how we judge the correct answer. In many cases, we simply cite an authoritative data source, such as a well-known database. In other cases, we provide details about how we would evaluate borderline cases. But ultimately, the INFER Team has sole discretion to decide what the correct answer is.


How often should I submit forecasts and how much time should I spend each time I submit a new forecast?

Each participant should submit and update forecasts several times per month, though we encourage more active engagement. Participants should expect to spend several minutes every week on the site reading through new questions and updating forecasts. Updating is highly encouraged: the most accurate forecasters tend to update their predictions more often than the average forecaster.


What should I include in my forecasting rationales?

Rationales should be short explanations that provide context to others about why you forecasted a certain way. You may share considerations that significantly affected your forecast, as well as links to any sources you found useful and that may be helpful to others. 

We consider a forecast rationale to be of high-quality for the purposes of the INFER program when it includes a clear, succinct summary of the forecaster's perspective indicating some level of external research. If you choose to cite research, be sure you are summarizing and stay on topic. Please refrain from pasting in lengthy or irrelevant excerpts from sources (e.g., news articles, AI tools, etc) as that defeats the intent of this community. See our policy for using AI tools like Chat-GPT in forecasts to learn more about this topic. 


Can I withdraw my forecast from a question?

There is no way to delete a forecast, but you can submit another forecast to reflect how you've changed your mind. If you forecast more than once in one day, only your last forecast of the day will count toward your accuracy score. You may also quit a question so that your forecasts will only be scored up until the day you quit. You will not receive scores for the days after you quit the question. To quit a question, visit the question you want to quit, click the gear dropdown in the upper left, and click "Quit Question."


I have a specific question regarding a forecast question, how can I get more specifics before I make a forecast?

If you need any clarifications, first read the Information box underneath the question. If you still need clarification, you can submit a New Clarification request directly from the settings within the question page. If you have any issues submitting a clarification request, send an email to [email protected].


What is INFER’s position if I encounter a question where I believe I need access to paid data to make an informed forecast?

All forecast questions on INFER can be answered using open-source information to come up with your best-estimate probability forecast. Where possible, INFER will provide historical or other relevant data within a question’s background section. If additional data is not readily available to you (e.g. it is behind a paywall), we ask that you make your best judgment using what is available. All questions released on INFER have been determined by our team to be important for generating signals for our research priorities. However, you may choose not to forecast in a question in which you feel you cannot make an informed judgment.

Back to Top

Forecasting Accuracy


What does it mean to resolve a forecast question?

When we know the correct answer to a question (e.g., when an event occurs), we can "resolve" the question. An INFER site Administrator will resolve a question, which then allows the site to measure and provide accuracy scores.


How are my forecasts scored for accuracy?

The INFER site uses a measure for accuracy that is also used by weather forecasters, called the Brier Score (the lower your score, the better). This article explains how we calculate the Brier Score.

To measure how accurate you are compared to the crowd, INFER uses an Accuracy Score (or Relative Brier Score). This score is based on the Brier Score, and is generally what shows up on the site leaderboards. This article explains how we calculate Relative Brier Scores.


How can I know my accuracy score?

You can find your cumulative Accuracy Score (or Relative Brier Score) listed in My Profile or on any of the site’s leaderboards. Please note that INFER forecasters will not have Accuracy Scores until a forecast question has been resolved.

Back to Top

Teams

How do I create a team?

  • Click the Create a Team link under “My Team” on the INFER navigation menu. If you are already a member of a team,  there will be an option to Switch Teams or Create a new team.
  • Enter a team name and the INFER usernames or email addresses (if they are not yet signed up for INFER) of all team members you wish to invite.
  • Submit your team member invitations – and remember, you can always add more teammates later.
  • You can view the invitation status of each team member once they are invited. Be sure to personally follow up with any team members who have not yet accepted your invitation to join the team.

Once you have an approved team, you will be able to interact with members of your team in different areas of the site, for example:

  • Go to “My team” on the top navigation menu to view and add other users on your team, their activity, and start general discussions.
  • Within any Question page (below the forecasting interface), navigate to the “My team” tab to start a discussion between only other members of your team and compare your team’s aggregate consensus to the crowd forecast.
  • Click on Leaderboards in the top navigation menu to see team rankings, which are updated automatically anytime a question is scored.
Back to Top

Forecaster Perks & Benefits


I hear that INFER offers rewards from time to time. How can I earn rewards?

INFER offers periodic participation challenges and monetary rewards for active forecasters. Past reward challenges have included forecasting on specific questions and topics and forecasting several times within a specified period. Check out all of INFER’s active reward challenges here.

Additionally, INFER hosts periodic forecasting tournaments where participants can compete to earn prizes.  


What is the Pro Forecaster Program? And, how can I become a Pro?

The INFER Pros are a team of paid forecasters who dedicate time to forecasting on INFER every month. Pro Forecasters participate in training, collaborate on smaller subteams with other Pros, and are invited to exclusive events. To learn more about the program, please visit the Pro Forecaster overview page.


What is the INFER Team Lead Program? And, how can I become a Team Lead?

INFER Team Leads help to build the INFER community and diversify perspectives by recruiting and engaging a team of at least 10 active forecasters each month. Team Leads are compensated each month for engaging their team and ensuring member participation. To learn more about the role and how to apply, please visit the INFER Team Lead overview page


Does INFER offer forecasting training?

We will soon be releasing a forecasting training curriculum. You can read more about it on the blog.

Back to Top

Privacy & Etiquette


What are INFER’s policies for participant privacy?

INFER is committed to protecting participant privacy consistent with the goals of the site as outlined in the Consent Form.


Do you have a content moderation or etiquette policy?

Our Etiquette Policy can be summarized as “Be kind, respect others, and stay on topic.” Inflammatory comments, personal attacks, vulgar or offensive labels, marketing efforts (“spam”), repeated, overt political advocacy, and aggressive behavior or harassment of any kind are all prohibited. If you believe that a participant’s comment violates this policy, you may flag it by clicking “Flag” below their comment or email us at [email protected]. The INFER team resolves violations of the Etiquette Policy on a case-by-case basis, and we reserve the right to disqualify offenders from rewards or other opportunities, and deactivate the accounts of offenders. This policy applies to other forums that are INFER-related (e.g., email, meetups, webinars, or INFER Discord server).


What is INFER’s policy on using ChatGPT or other AI models to help generate forecasts and rationales?

AI tools can be useful for initial research into forecasting topics and can help provide ideas for developing your own perspective. INFER’s policy is that these tools should be used responsibly on our site for research purposes, to further promote discussion and individual analysis, and with proper attribution.

Proper attribution should include the AI model used to generate any part of your content. For example: “The following ideas are summarized from ChatGPT: [insert content].”

Since INFER provides government stakeholders with frequent reports that include rationales from forecasters, we expect those rationales to be original and thoughtful analysis. Copying and pasting content directly from AI models into your rationales without proper attribution and without your own analysis and interpretation may be copyright infringement and impedes the quality and credibility that the INFER community upholds in our human-driven insights. Posts that appear to be directly authored by AI tools such as ChatGPT will be in violation of our policy. Site administrators may remove content, and repeated violations by a forecaster may result in removal from the site.

Back to Top
Files
Tip: Mention someone by typing @username