The workflow before
Research analysts and portfolio teams gathered external data themselves: regulatory filings, company investor relations pages, news and alternative data. Collection was manual, so coverage stopped at about 50 sources and reached roughly 20% of what teams needed. Analysts spent 30 to 40% of their week gathering and cleaning data instead of analyzing it, and information arrived daily or weekly rather than when it was published.
What we built
We built an automated collection system on AWS that crawls more than 1,000 sources: SEC EDGAR and international filings, company sites, news and alternative data. Scrapy handles standard pages, Selenium renders JavaScript-heavy sites, and API clients pull from providers that offer them. Everything streams through Kafka.
NLP models extract companies, people, transactions and events from each document, classify it and score its relevance. Entity resolution matches every mention to BlackRock's security master by ISIN, CUSIP and ticker, with low-confidence matches sent to a human reviewer.
Clean data lands in an S3 data lake and Snowflake, where analysts, data scientists and the Aladdin team query it. Change detection spots when a source site changes structure, so crawlers are fixed before data goes missing.
Before and after
| Measure | Before | After |
|---|---|---|
| Sources covered | 50, collected by hand | 1,000+, automated |
| Analyst time on data collection | 30 to 40% | Under 5% |
| Coverage of needed sources | About 20% | 95% |
| Data freshness | Daily or weekly | Near real time |
What's next
With 1,000+ sources resolved to the security master, the next step is L3: agents that monitor holdings for material events, draft the summary, and route only the items that need an analyst's judgment.
