Metrics/Evaluation 2.0, Part One
Posted On: August 27, 2019
An update from FMC Senior Advisor Darlene Wolnik on the ongoing development of its Metrics product and on FMC’s work in providing overall evaluation and technical assistance to markets and networks.
Background: FMC and evaluation
Across the last decade, the increased use of data by market organizations to make decisions and to enlist new partners has been a constant source of interest at FMC. That trend has led to those organizations wanting more control over the choice of data points, rather than just serving as the sites where data was to be collected (and used) by others.
This development can likely be traced to:
1) the increase in embedded permanent programs at markets designed to increase the number and diversity of uses and users;
2) that many are determined to take back the language of community food/markets before it is entirely co-opted by the industrial food sector;
3) meeting the challenge in keeping hard-fought locations and to reduce other snarls that arise from their municipality and regional governments misunderstanding the role of a community farmers market;
4) making decisions on market day and at the organizational and network level based on data from many, rather than only listening to the “squeaky wheel,” or waiting until the crisis arises.
FMC has directly supported this emerging culture of data-driven markets, starting with its partnership with Market Umbrella in its trans•act fellowship in 2008, in its many pilots and partnerships with food system organizations and researchers since, and led to its development of Metrics.
Metrics allows markets to control the choice and use of data, and allowing them to easily share it with their partners; organizations can now purchase annual subscriptions to the data management and reporting functions. FMC’s Metrics team also offers direct technical assistance to customers through an online Help Desk system, email and phone support, and through scheduled monthly calls (“Clinics”) as well as a monitored group page. Added technical assistance and customized webinars about general data collection or network strategies can also be purchased by users.
Why Metrics?
For its part, FMC has focused on two main drivers in designing Metrics:
- our clear mission to support and directly advocate for market operators. That markets are a variety of types and levels of capacity is something we constantly keep in mind to not offer a one-size-fits-all response.
- to expend effort to build new tools and resources only when there is an obvious need and by letting our field do what it does best: to have the primary learning environment remain with the grassroots, front-line experience of market operators.
In this case, even with the excellent resources and examples already out there (including RMA, SEED, and FM Tracks among many others) markets told us there wasn’t an online tool for market organizations that was flexible or user-friendly enough, or one that offered contextual, dynamic and disciplined reports. (Here is more of our thinking about existing tools and lessons learned as outlined in our December 2018 JAFSCD article.)
That, in a nutshell, was the reasoning behind Farmers Market Metrics. It isn’t perfect or even done by any means. But the hundreds of markets across 25 states now entering data into the system show that we are succeeding in meeting our main drivers.
Networks
After market organizations, state and network organizations were the next primary group that we added functionality for in Metrics. Although in practicality, the desire for aggregated data has always been a primary outcome for FMC and with those networks; it’s just that we didn’t design Metrics for network users to be the primary account holders.
That sometimes frustrates network leaders, which has certainly led to many frank conversations with them and within FMC as to how to better serve their data needs. For many markets and some networks and research partners, Google Forms /spreadsheets are plenty good enough to be able to collect and control the data as they see fit. We certainly understand that.
As a matter of fact, in the first days of the Metrics network buildout, FMC worked on just such a spreadsheet system via a request from our first network partner. Interestingly, the end result was that the network didn’t find it that useful, as it still required a lot of oversight and entry issues for the market organization which were difficult and too time-consuming for many to master. Still, that lesson taught us that we needed to keep testing approaches and creating resources for a wide group of users, in Metrics and beyond it. As a result, spreadsheets remain a key component of Metrics and uploading bulk data via spreadsheets is also doable for Metrics users. (Not yet super easy but doable!) We’ve also been discussing how the FMC team can keep learning more through analyzing data using offline spreadsheets of data shared by markets and networks.
As that story above (hopefully) shows, in working with networks we have learned a great deal about aggregated data and what other networks are hoping to get out of Metrics.
The first reason that networks are telling us that they need Metrics is to do what FMC is also deeply interested in:
-to support what they can see is the emerging culture of data-driven farmers market organizations.
-And then via FMC’s advocacy work, we can see the other primary reason networks are eager to use Metrics is that much of the data use so far has been driven by network reports showing off aggregated data.
Virginia Farmers Market Association and Austin area farmers markets, in partnership with the Texas Center for Local Food and the City of Austin, are two recent and excellent examples of that. Check out their wonderful reports and press releases in support of National Farmers Market week 2019.
But even though networks are currently using the data at a higher rate, we still believe the use of data by individual markets for both internal and external needs is the role that Metrics fills best and we believe is the future of market data.
Because of all those lessons, we definitely see the need for a much more detailed and intentional on-boarding process for networks, and as such will be taking on new networks on a case-by-case basis as we define and design that process. Check with us if you want to become a new network; we’d like to understand more of your needs first.
And because data aggregation across market organizations is infinitely more complex, we are still deciding on what the future development of network reporting within Metrics will look like, and are likely to spend some effort on offering in-house design services for reporting that cannot be done in Metrics easily. It may even become necessary to roll back some of the network reporting functions that are becoming too complex. The good news is that market organizations can continue to download excel spreadsheets of this data and share it with their networks as needed. It also tells us that data in spreadsheet form and dashboards are likely to be the most useful for networks. As that story above (hopefully) shows, in working with networks we have learned a great deal about aggregated data and what other networks are hoping to get out of Metrics. The main reason that networks are telling us that they need Metrics is to do what FMC is also deeply interested in: to support what is the emerging culture of data-driven farmers market organizations.
And via FMC’s advocacy work, we appreciate how much of the data use that we have noted so far has been driven by network reports showing off aggregated data.
Virginia Farmers Market Association and Austin area farmers markets, in partnership with the Texas Center for Local Food and the City of Austin are two recent excellent examples of that. Check out their wonderful reports and press releases in support of National Farmers Market week 2019.
But even though networks are using the data at a higher rate, we still believe the use of data by individual markets for both internal and external needs is the role that Metrics fills best.
That is partly because the work happens at the grassroots level, but also because markets tell us and their network partners that we all have to respect the diversity of approaches and capacity levels by markets across the U.S.
