3.0 Methods & Implementation
In Units 1 and 2, you selected the metrics that best fit your farmers market’s current goals. Next, think about the specific data that will be collected for those metrics. Farmers Market Metrics focuses on numeric, or quantitative data, collected using one of three methods: document review, surveys or observation.
Primary and Secondary Data
Market managers, volunteers, and organizers may not think of themselves as “data collectors.” Nonetheless, when a market counts visitors, collects sales figures, or records volunteer hours, it is collecting and creating primary data. Vendor applications, records of card transactions, and visitor count sheets are all examples of tools that collect primary data.
Secondary data, on the other hand, has already been collected, analyzed, and (usually) published somewhere. Secondary data allows you to compare your primary data to other data in your state, region, or even nationally. Three well-regarded sources of secondary data that could provide relevant information to your market are the USDA’s Census of Agriculture, the Food Environment Atlas, and the American Community Survey. Your city or county may also have commissioned studies or have a Community Food Assessment which may help you understand demographics, land use, or nutrition assistance participation in your area. Ask around to see if there are reports that you can review. In addition, income, consumer spending, and economic multiplier data is available by region from the Bureau of Economic Analysis. The USDA Economic Research Service even has national data on food purchasing trends. Searching for the appropriate databases may be a place that researchers can assist your evaluation project. That data can be compared to the data you collect. Farmers Market Metrics uses some secondary data in the reporting. For instance, for the food donated metric, this is below the metric when it is used:
99% of food banks reported seeing more first time users in the last year. As you can see, use of secondary data can make your own data very powerful.
Three Methods for Data Collection at Farmers Markets
In FMM, we focus on three general methods for collecting data at farmers markets. Below are the methods listed with a few examples of how markets use them.
- + Document Review – using administrative documents, or those created during the completion of market activities, to pull out relevant data points. A good example of this is using the merchant report from an EBT processor to determine the day’s total SNAP sales.
- + Observation – taking note (usually via a tally) of products, people and conditions to keep records of weather, special events, visitor counts.
- + Surveys –asking questions of customers or vendors through questionnaires, to determine vendor demographics, and customer desires or activities.
These three methods were selected for FMM because they are relatively cost-effective, practical, and gather the same information that farmers markets across the country are already collecting.
To begin data collection, plan the dates well ahead of time to organize the labor and resources needed. Weather or other issues may arise which may make it seem better to cancel – but if it all possible, keep that agreed upon date. Alert your vendors and shoppers to the collection dates using the market website, Facebook page and signs at the market leading up to collection days.
If volunteers are needed, add that request to chalked market signs, the market Facebook page and in email newsletters weeks ahead to increase chances of encouraging market enthusiasts to help with collection. Reach out to stakeholder organizations to see if they can help with staffing for the collection. There is a separate document on Data Collection Tips to look through for ideas as to staffing your data collection team.
This unit’s collection instructions in Part 2 are based on markets collecting survey data on a minimum of two days per season, with four days of surveying strongly encouraged. The thinking is that markets with an average of a 25-week season need at least two sample days of data and should be able to gather the resources at least twice to do it correctly. Year-round markets should schedule data collection just as a one-season market does, adding more rounds of data collection using the same intervals (i.e. the 10th and 16th market week of the winter season). If the data is meant to represent the entire year, collecting data only in the summer season for example could overinflate the data or if only collected in the slower season, may underreport the market’s impact. In other words, don’t think of data collection as never-ending, but as campaigns with a kickoff, check-in, and a wrap up at the end of each campaign.
Consider following or adapting the example schedule presented in Figure 3.1 below. The bolded days are the primary days for two days of collection. However, if you have the data collection team available and would like more precise data, the other two dates should be added.
Counting Days: On the 4th, 10th, 16th and 22nd market days of the season
Surveys: On the 4th, 10th, 16th and 22nd market days of the season
Observation: On the 6th, 12th, 18th and 24th market days of the season
Ideally surveys should be conducted on the same day as counting visitors. If two seasons of data collection are desired, repeat the dates above for each season.
Figure 3.1 Farmers Market Metrics Data Collection Schedule
Collection Method: Document Review
Document Review includes two types of market methods: Operations Research and Secondary Research. Operations Research makes use of the documents and data collected during normal farmers market management activities.
One of your first steps with Operations Research is to take stock of what information your market is already collecting, either for your own use or for any farmers market association, public benefit program (i.e., FMNP, SNAP EBT), grant, or market sponsor. Much of this information is already in your vendor applications, market day forms or logs used to track vendor fees, vendor attendance, payment processing statements, etc. Hopefully, collecting and storing at least some of this information is already a routine part of your market operations.
As you consider your market operations research, you should:
- Compile a list of all the operational documents your market already has on file.
- Review the documents and inventory all the information or data that you already have. What’s missing? What would you like to add next season? What condition is the data in? Is it complete, or are there missing variables?
- Modify future market operations documents to get the information you need.
Organizing your information and making it easy to access are also critical. This is especially true when there is a change in the primary market manager or if market information is being stored on personal computers. Once your market’s operations data collection is routine, then your energy can be dedicated to keeping it up-to-date and compiling it in a useful way.
Secondary Research searches secondary databases which were described above this section. This is a great place to ask for assistance from those partner organizations that have conducted research previously. It is not necessary to your use of Farmers Market Metrics, but might be helpful, especially if you are planning to present your data to university or public health funders or partners. Farmers Market Coalition maintains a set of Talking Points and other citations culled from market reports within its Advocacy Toolkit that can be used to find relevant research.
Collection Method: Observation
Observation is a way of gathering data by counting people or products, documenting behavior, events, or physical characteristics in their natural setting. Observations can be “overt,” meaning everyone knows they are being observed, or it can be “covert,” meaning no one knows they are being observed.
The benefit of covert observation is that people are more likely to behave naturally if they do not know they are being observed. However, covert observation methods raise potential ethical problems that most markets want to avoid. That’s why markets conduct overt observation, and should post a sign letting everyone know about any planned observation or data collection.
This section covers two types of market methods that are considered Observation:
- + Tallies or counts of people and products;
- + Observing behavior or clusters of activity.
The advantages of observation methods include:
- Tallies of items like market goods or the number of recipes taken, for example, can be gathered directly by the market staff, volunteers, or the data team, rather than asking vendors to submit data;
- Observers can note what doesn’t happen, which is also valuable information. “The seating near the event was not used.”
Counts of People or Products
Tracking the numbers of market visitors, vendors, and products is critical to knowing how well a farmers market is doing. You may use this data to be able to answer questions such as:
- What are our average sales per shopper?
- On average, how many visitors come to our market? Is it growing? Declining?
- What are the patterns over the market season?
To answer these questions, you need to have good data collected in the same way each time.
First, you must define the category that will be counted. For example, we recommend choosing “adult visitor” as your category or unit of data for most of the tally metrics, except for those that specifically concern children’s educational activities. Some markets choose to count “anyone with a wallet” (including teenagers). Whatever the category or definition of your unit, make sure you spell it out to your data collectors and in your final report.
The method for counting visitors in this unit is an entry sample or a timed entry count, which means that every adult visitor is counted as they enter the market for predetermined time periods. The FMM materials recommend using 20-minute entry counts and includes collection instrument templates for that collection method. If you would like to know more about visitor counts and how Farmers Market Coalition landed on the 20-minute count, check out the article “Counting Visitors at Markets” resource in the FMC Resource Library.
Greater accuracy is achieved by counting everyone entering the market, known by some as a full count. If trained staff is available to do a full count, by all means do so – it will generate more precise numbers and make related computations more precise as well. But be aware: the full count is still considered a sample of overall attendance if not done every market day. It can be labor intensive, and may be difficult to manage for markets with many entrances and large attendance. Farmers Market Coalition has been collecting experiences from markets and researchers about different methods of counting visitors located in the Resource Library.
When counting products for metrics such as “Average number of SNAP-eligible goods available per market day” the market needs to be clear with those collecting how to handle duplicates or different varieties of the same product before they begin their data collection. It is also helpful to complete product tallies as early as possible in the market day before some items sell out. FMM includes a collection instrument template for this data collection and recommends counting each product (tomato) at every vendor’s table, and to count multiple displays within one vendor’s set up only once. The definition of the data to be collected for each metric is based on the way the data is to be used by the market or its partners. For example, “Average number of SNAP-eligible goods available per market day” is meant to be a indication of how well the market vendors can handle an influx of SNAP shoppers so counting how many places a shopper can buy that product is helpful to that assessment.
Tips for Implementation
- Avoid conducting counts during the opening market day, the last few market days of the season, or on market event days. In other words, choose the most representative dates of your season(s). If you have events every market day, be sure to note what was happening on the day of the count.
- Communicate the news about your designated counting or survey days as early as possible to all members of your farmers market community (vendors, volunteers, board members).
- In the weeks before a counting day, walk through the process of what will happen on those days with vendors. If you are unsure about the process you have set up, consider conducting a trial count on the first or second market day, to give yourself a chance to work out any kinks.
Collection Method: Surveys
We focus on three types of surveys that farmers markets often use. They include questionnaires (also called self-administered surveys or sales slips), intercept (face-to-face or interview) surveys, and dot surveys. Farmers Market Metrics does not currently use Dot Surveys as the preferred collection method as the metrics included have closely worded questions or contain sensitive information such as demographic information or sales data.
Samples can be randomized (asking every second person that walks by), self-selected (having a link to a survey), or pre-selected (asking all vendors or incentive shoppers to fill one out). Regardless of your selection technique, be sure to note it in your data files, so it is part of your methods.
The size of the sample (how many people take the survey) is also important. There is always a question of how large a sample you need to produce “valid” results. We provide a sample size template in Part 2 as well as a link to a sample size calculator. When you do surveys, you will also hear terms such as margin of error and confidence level. Those terms are meant to calculate how well the sample represents the population. For example, a survey may have a margin of error of plus or minus 5 percent at a 95 percent level of confidence. These terms simply mean that if the survey were conducted 100 times, within 5% above or below, the answers would match the percentages reported in 95 of the 100 surveys. In other words, if 100 more people were surveyed, the percentage of those that come weekly would not differ by more than 5% than the first 100 people surveyed.
It is important to note that the margin of error decreases as the sample size increases, but only to a point. A very small sample, such as 50 respondents, has about a 14 percent margin of error while a sample of 1,000 has a margin of error of 3 percent. This is important in market surveys as smaller markets could have a large margin of error if they set too low quota of surveys on collection day. But it also helps larger markets, as the number of surveys to collect does not continue to grow ever larger.
To reduce over or under sampling, we recommend that markets do their best to follow the sample size or use the sample calculator to get the appropriate number of surveys to collect.
Markets can certainly choose different levels, based on their partners’ and their own analysis needs. As for the response rate, this is usually very high for market surveys, as most of those approached at markets are happy to help. However, if you are surveying outside population (for example, at a community center) or your vendors it may be more difficult to get completed surveys. Have your data collectors keep track of how many refusals they receive.
When you are doing surveys for the season, then estimate your total population by adding up all of the market days visitor attendance estimates during that season. (20 market days @ an avg. of 200 visitors per day (4,000 visitors) divided by half means around 2000 would be the total population on which to base your sample size. That is because of those 4000 visitors, it is reasonable for most markets to expect that at least half of them are regularly returning visitors. If you would like to know the actual number of returning visitors in your market for future years calculations, you can add that data point to your visitor survey, asking people how often they attend the market.) Use this number to find the survey number needed, then divide that number by the number of days that you will have a survey team. If you have 100 surveys to collect over 2 survey days, then 50 per day and 25 per hour will be your collectors goal. Of course you want to make sure that the completed surveys are spaced out across the entire day so that you get a representative number of early, midday and late shopper responses.
Whew! That’s a lot of detail about percentages and statistics. The most important thing to know is in order to make sure you get enough responses, take the calculation seriously and either ask a research partner, use the sample size suggested here, or use an online sample calculator.
For all surveys, think through your research question clearly. Keep in mind that yes/no, multiple choice and numeric answers are far more straightforward to clean, analyze and report. For FMM metrics, survey questions and resources for collection are included.
Three more important tips for conducting successful surveys are listed below:
- + Test your final survey with your friends, colleagues, or some other “pilot” group to work out the bugs;
- + Keep your surveys as short as possible. This is easier said than done, but short surveys increase your response rates (more people will complete them) and are easier to clean, analyze, and report. Try to keep the entire exchange under 2 minutes; remember; people are there to work or to shop, not to answer dozens of questions. Whenever you are not sure, ask yourself if the added information is necessary to collect that year. Often markets overload their surveys and burn out their collectors and make the data entry and management very difficult for themselves. Keep in mind you can always ask other questions in another round of data collection later.
- + In some cases, offering an incentive to respondents can help increase your response rate. At a farmers market easy incentives include free tokens, free coffee or tea, or even free produce.
A questionnaire or self-administered survey is a useful way to gather data while preserving a degree of privacy.
Written questionnaires can take the form of paper or online questionnaires. If you have current emails for the population you want to survey, an online questionnaire can enable you to collect information from a large number of people in a short period of time and in a cost-effective way.
The results of an online questionnaire can usually be obtained quickly and easily through online services such as Survey Monkey or Google Forms. Be sure to check what each version of software allows; in the free versions, these companies often limit how many responses you can collect and how the data can be exported. These online services also offer paid options for larger samples.
Data obtained through questionnaires does have limitations: some respondents may not recall answers to every question or may not always be truthful. Additionally, questionnaires may not capture all the desired information, depending on the length of and the number of questions posed by the researcher.
Many markets are hesitant to ask their vendors sensitive information about sales or other business data, which is a perfectly valid concern. However, since markets are tasked with advocating for their community with policy makers and others, sales data is key. To assuage their concerns, it is important to safeguard the information and to only ask what and when necessary. FMM will continue to research methods for collection of sensitive information and to offer additional tips and templates for collecting this data.
Intercept or face-to-face surveys capture real-time information by approaching market visitors on site and inviting them to complete a survey. Intercept surveys may allow the data collector to reach the desired number of surveys quickly over the course of the market day.
Intercept surveys also have some drawbacks:
- + Visitors may not like being interrupted in their usual activity at the market.
- + Intercept surveys require training of data collectors.
- + The market may need to recruit many volunteers or paid data collectors.
- + Weather may also be a factor in survey success, since reliable results depend on surveying a relatively large percentage of the shopper population.
Dot surveys are popular instruments for collecting information from market visitors about simple “yes or no” or multiple choice questions in which only one response is possible. A small number of questions (1-3) are displayed on sheets of paper on easels or flipcharts. Respondents indicate their responses using colorful, stick-on dots during their visit to the market. Dot surveys are a core part of Rapid Market Assessments (RMA), along with visitor counts and constructive comments and observations from the RMA team.
The benefits of using dot surveys include the fun, interactive atmosphere they contribute to the market and their typically high response rate. The main limitation of dot surveys is the time required to recruit volunteers and figure out the right questions and responses that work well in a dot survey, as well having someone on hand ready to hand out the dots to enough people, and to change the sheets of paper regularly. This type of survey may not be appropriate for sensitive or personal data, and like other intercept surveys, may also be negatively impacted by bad weather. Answers can also be influenced by prestige bias, meaning that respondents can be influenced by previous answers clustered in one area.
Tips for Successful Data Collection Days
For many metrics, data is initially obtained on paper, whether it’s a physical slip of paper handed in by a vendor at the end of the day, a visitor count tally sheet, or an intercept survey on which members of the data collection team wrote visitor responses using a clipboard as they walked around the market. The resources for FMM collection are discussed in Unit 3, Part 2.
Keeping track of all these pieces of paper can itself be challenging, so here are a few tips for keeping track of surveys and tally sheets:
- Make sure the date received is recorded on every data collection form, in the same place on each instrument.
- Assign a unique identifier to each completed survey or tally, using incremental numbers like 001, 002, and 003. Write this number on the corner of each paper survey questionnaire, and enter that same number in the column labeled “ID#” followed by the data for that questionnaire. Later, this will allow members of the research team to find a particular questionnaire even if identifying elements (name or contact information) are removed from your electronic database.
On data collection day:
- Have physical maps of the market on a clipboard or a table in a central location. If there is not a market booth or table normally, it would be helpful to set one up for the day with bins for the data collectors to store their personal items (have them leave expensive items in their car or at home) and to keep water and extra supplies and even chairs for between “shifts” if applicable.
- Start the day of data collection by explaining the breakdown of the day, including sketching out where everyone is to be and at what time each person needs to be there. It might be helpful to write everyone’s name on the map in the location they are assigned with their mobile phone number if applicable. Have everyone introduce himself or herself.
- When handing the collectors their interview or observation sheets, mark their name and their goal for completion for that hour or their specific time-period on the top sheet or on a post-it note. Ask them to come back to the booth or arrange to meet them at a specified time at the end of that time-period. All collectors should have 5 minutes as needed for a break or a 20-30 minute break for those collectors working the entire market day.
- Give everyone a name and a phone number to call if they have an issue, even if that person will be checking in regularly. If possible, introduce them to the vendor nearest to their station.
- At the end of the first collection timespan, ask collectors how things went, ask them to describe one of their data collections, and ask them for any ideas to make it easier or more fluid the next time.
- If there is time, do one or two rounds of role-playing with the market leader acting as the data collector and if possible, a regular volunteer as the respondent.
- Assure the collectors that this will go well and be easy work. Offer some sample language for intercept or dot surveys, such as, “Do you have a minute to help the market?” and give a simple explanation to use when questioned by visitors or vendors, especially for observers and tally collectors, such as, “We’re gathering information to help the market.”
- Assign one person to look through completed surveys and check the tallies throughout the day. Ideally, this person will also do the data entry, which will be explained in Unit 4.
These training materials were created with support from the United States Department of Agriculture’s (USDA) Farmers Market Promotion Program (FMPP) FL-326-G, and reference research in process as part of the USDA’s Agriculture and Food Research Initiative (AFRI) 2014-68006-21857.
Created by Darlene Wolnik and Jennifer Cheek with support from Sara Padilla, Stacy Miller and a panel of expert advisors. Special thanks to Colleen Donovan for her review and contributions.