Ashleigh Chapman
At Engage Together, we help communities strengthen their efforts to end and prevent human trafficking, and to care well for all those impacted by it. An essential component of our work is completing a comprehensive assessment of every program and resource that exists in a community (or an entire state) to support survivors and at-risk populations.
We begin that process by first searching the internet for all relevant programs, efforts, and resources, community by community. We break the initial phase of this research down into two actions. First, searching the web to gather an accurate and comprehensive list of all active efforts within the region. Then second, meticulously collecting data on each of those efforts, such as websites, addresses, contact numbers, partners, program details, and much more. In addition to our own team members that work extensively on this process, we often also hire a dozen interns per region to support this part of the research effort.
Alongside our partner, Pomerol, we tackled searching the web using another search API. However, that left us with a (very) long list of potentially relevant websites. Instead of going down a path of attempting to code scraping logic for each and every website, what we needed instead was a generalized method of scraping data that could tackle any website that appeared in the search and was relevant to the project. And instead of trying to build this ourselves, we chose to leverage Firecrawl.
We were actually in the process of architecting a solution when Firecrawl released its /extract API. This took all the custom work out of it. All that was left for our team to build was an end-to-end process that could handle the large amount of asynchronous calls and storage. The step of marrying the search API and Firecrawl’s API was the simplest solution for us, and perfect for our use case.
If we had to stop using Firecrawl, we’d miss /extract the most, as it is a core component of our scraping process. The integration went smoothly, and we didn’t even need to use any of the support made available on the forums or Slack. The documentation itself was sufficient.
In summary, we are thrilled to have Firecrawl serve as a robot intern on our team that helps us collect critical information from thousands of websites in a fraction of the time it used to take us to do it all manually. Which means we can now focus the bulk of our time on enriching and analyzing the data, not in finding it in the first place. Thank you!
data from the web